Oct 30 17:58:11.060190 ip-10-0-143-94 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:58:11.060202 ip-10-0-143-94 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:58:11.060209 ip-10-0-143-94 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:58:11.060472 ip-10-0-143-94 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:58:22.244382 ip-10-0-143-94 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:58:22.244394 ip-10-0-143-94 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bc6edec1a2934972b1e12f7f94bbe004 -- Oct 30 18:00:06.993875 ip-10-0-143-94 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:00:07.390110 ip-10-0-143-94 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:07.390110 ip-10-0-143-94 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:00:07.390110 ip-10-0-143-94 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:07.390110 ip-10-0-143-94 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:07.395875 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395794 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.395875 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395874 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.395875 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395879 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395883 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395886 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395888 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395891 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395900 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395903 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395906 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395910 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395912 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395916 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395918 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395921 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395924 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395926 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395930 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395933 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395936 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395939 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395941 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.395969 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395944 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395946 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395949 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395951 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395954 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395957 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395959 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395962 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395965 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395967 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395970 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395973 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395975 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395978 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395982 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395984 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395987 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395989 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395992 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.396483 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395995 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.395998 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396000 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396003 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396005 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396008 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396010 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396012 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396015 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396017 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396020 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396022 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396025 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396027 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396031 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396035 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396037 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396040 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396043 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396045 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.397143 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396049 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396052 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396055 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396058 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396061 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396064 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396066 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396069 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396072 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396074 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396077 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396079 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396082 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396085 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396088 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396090 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396093 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396096 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396098 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396101 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.397836 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396104 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396107 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396109 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396112 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396114 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396117 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396120 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396122 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396124 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396127 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396130 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396132 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396135 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396138 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396143 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396147 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396150 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396153 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396156 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.398393 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396158 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.398879 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396161 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.398879 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396163 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.398879 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396166 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.398879 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396168 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.398879 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.396171 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.399247 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399229 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399248 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399252 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399256 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399259 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399263 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399267 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399270 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399273 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399276 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399279 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.399277 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399282 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399285 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399288 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399290 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399293 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399296 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399298 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399301 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399304 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399306 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399309 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399312 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399315 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399317 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399321 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399325 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399328 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399331 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399334 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.399555 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399337 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399340 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399342 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399345 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399347 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399350 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399353 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399356 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399358 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399360 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399363 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399366 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399368 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399371 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399374 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399377 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399380 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399383 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399386 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399388 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.400034 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399391 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399394 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399397 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399399 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399402 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399404 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399407 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399409 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399412 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399415 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399417 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399420 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399423 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399425 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399428 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399430 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399433 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399435 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399438 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.400641 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399440 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399443 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399446 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399448 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399452 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399455 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399458 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399460 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399464 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399467 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399469 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399472 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399474 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399477 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399480 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399483 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399486 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399488 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399491 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399494 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.401102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399496 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399499 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399501 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399504 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399507 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399510 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399512 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399515 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399518 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399520 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399523 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399525 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399528 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399530 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399533 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399535 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.399538 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400138 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400146 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400150 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.401572 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400154 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400158 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400161 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400163 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400166 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400169 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400178 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400181 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400184 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400187 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400189 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400192 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400195 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400197 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400200 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400203 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400205 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400208 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400211 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.402066 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400213 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400216 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400219 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400221 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400224 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400227 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400229 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400232 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400236 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400240 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400244 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400247 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400249 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400253 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400255 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400258 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400261 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400263 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400266 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.402518 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400268 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400272 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400274 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400277 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400280 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400282 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400285 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400287 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400290 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400292 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400295 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400297 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400300 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400302 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400305 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400307 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400310 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400312 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400315 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400317 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.402990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400320 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400322 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400325 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400328 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400330 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400333 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400336 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400339 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400341 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400344 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400347 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400349 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400352 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400354 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400358 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400360 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400363 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400365 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400368 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400371 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.403503 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400375 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400379 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400382 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400385 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400388 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400391 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400394 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400397 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400399 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400402 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400405 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400409 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400411 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400414 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400417 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400419 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400422 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400425 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400428 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400431 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.403996 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400434 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400437 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400439 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400443 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.400446 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400538 2569 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400545 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400552 2569 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400557 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400561 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400564 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400568 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400572 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400576 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400579 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400582 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400585 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400610 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400614 2569 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400617 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400620 2569 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400624 2569 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400626 2569 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:00:07.404467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400630 2569 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400633 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400636 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400639 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400643 2569 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400647 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400650 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400653 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400658 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400663 2569 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400667 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400671 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400674 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400677 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400681 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400684 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400687 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400691 2569 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400694 2569 flags.go:64] FLAG: --enable-server="true" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400697 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400704 2569 flags.go:64] FLAG: --event-burst="100" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400708 2569 flags.go:64] FLAG: --event-qps="50" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400711 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400714 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:00:07.405027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400717 2569 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400721 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400724 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400727 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400730 2569 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400737 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400741 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400745 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400748 2569 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400751 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400754 2569 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400757 2569 flags.go:64] FLAG: --feature-gates="" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400761 2569 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400764 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400770 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400774 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400777 2569 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400780 2569 flags.go:64] FLAG: --help="false" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400783 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400786 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400789 2569 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400792 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400795 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:00:07.405633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400799 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400801 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400804 2569 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400808 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400811 2569 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400814 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400817 2569 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400820 2569 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400823 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400826 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400829 2569 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400832 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400834 2569 flags.go:64] FLAG: --lock-file="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400837 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400841 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400844 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400854 2569 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400857 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400860 2569 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400863 2569 flags.go:64] FLAG: --logging-format="text" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400866 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400869 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400872 2569 flags.go:64] FLAG: --manifest-url="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400875 2569 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400879 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:00:07.406184 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400882 2569 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400886 2569 flags.go:64] FLAG: --max-pods="110" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400889 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400892 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400895 2569 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400898 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400901 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400904 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400906 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400914 2569 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400917 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400920 2569 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400923 2569 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400926 2569 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400929 2569 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400932 2569 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400935 2569 flags.go:64] FLAG: --port="10250" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400938 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400941 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0681bd3f001fe1f10" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400944 2569 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400947 2569 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400952 2569 flags.go:64] FLAG: --register-node="true" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400955 2569 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400959 2569 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400962 2569 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400965 2569 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:00:07.406789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400968 2569 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400971 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400974 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400977 2569 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400980 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400983 2569 flags.go:64] FLAG: --runonce="false" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400986 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400989 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400992 2569 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400995 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.400998 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401001 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401004 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401007 2569 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401010 2569 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401013 2569 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401016 2569 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401019 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401023 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401026 2569 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401028 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401034 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401037 2569 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401039 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401043 2569 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:00:07.407470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401046 2569 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401049 2569 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401053 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401056 2569 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401060 2569 flags.go:64] FLAG: --v="2" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401064 2569 flags.go:64] FLAG: --version="false" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401069 2569 flags.go:64] FLAG: --vmodule="" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401073 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401076 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401212 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401216 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401219 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401221 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401224 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401227 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401230 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401233 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401235 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401238 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401241 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401243 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.408100 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401246 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401248 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401251 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401253 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401256 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401259 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401263 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401268 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401271 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401273 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401276 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401279 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401282 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401285 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401288 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401292 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401294 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401297 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401299 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401302 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.408618 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401305 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401308 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401310 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401313 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401316 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401318 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401321 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401323 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401327 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401331 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401334 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401337 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401339 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401342 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401344 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401347 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401350 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401352 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401355 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.409102 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401359 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401361 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401364 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401366 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401369 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401371 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401375 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401378 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401382 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401384 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401387 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401390 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401392 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401395 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401397 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401400 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401402 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401405 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401407 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401410 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.409558 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401412 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401415 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401417 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401420 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401422 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401425 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401428 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401430 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401433 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401436 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401438 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401441 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401444 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401447 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401449 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401451 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401454 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401456 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401460 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401463 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.410049 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401467 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401469 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401472 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401474 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401477 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401480 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401482 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401485 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401487 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401490 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401492 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401495 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401497 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401500 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401502 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.401528 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:07.410520 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401569 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401574 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401577 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401581 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401584 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401587 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401589 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401592 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401612 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401616 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401619 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401622 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401624 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401627 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401629 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401633 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401638 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401640 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401643 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401645 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.411021 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401648 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401651 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401653 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401656 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401658 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401661 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401663 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401666 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401669 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401671 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401674 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401677 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401679 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401682 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401684 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401687 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401689 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401692 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401695 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401697 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.411502 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401700 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401702 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401705 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401708 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401711 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401713 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401716 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401720 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401724 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401726 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401729 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401731 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401734 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401737 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401739 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401741 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401744 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401746 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401749 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.411990 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401752 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401754 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401757 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401759 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401763 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401766 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401768 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401770 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401773 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401775 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401778 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401780 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401783 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401786 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401788 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401791 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401795 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401799 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401802 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.412462 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401804 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401808 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401812 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401814 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401817 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401820 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401822 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401825 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401827 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401830 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401832 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401835 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401838 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401840 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401843 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401845 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401848 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401850 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401853 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.413110 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401858 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401862 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401865 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401868 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401870 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401873 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401875 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401878 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.401880 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.409332 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.409348 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.409363 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.409368 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409431 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409437 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409440 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409443 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409446 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409449 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.413690 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409452 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409455 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409458 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409461 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409464 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409467 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409469 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409472 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409475 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409477 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409480 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409494 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409498 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409501 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409504 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409508 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409511 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409514 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409517 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409520 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.414210 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409522 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409525 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409528 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409531 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409534 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409539 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409543 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409546 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409548 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409551 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409554 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409556 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409559 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409562 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409564 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409567 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409570 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409573 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409575 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.414706 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409578 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409581 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409584 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409586 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409589 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409591 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409612 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409615 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409618 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409620 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409623 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409626 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409629 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409632 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409634 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409637 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409640 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409642 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409646 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409648 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.415151 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409651 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409654 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409657 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409659 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409662 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409664 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409667 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409670 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409672 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409675 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409677 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409680 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409682 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409685 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409687 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409690 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409692 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409695 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409697 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.415721 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409700 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409704 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409707 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409710 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409713 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409716 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409718 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409721 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409723 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409726 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409729 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409732 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409735 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409738 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409740 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409743 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409752 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409755 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409758 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.416155 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409760 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409763 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409766 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.409796 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409954 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409960 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409964 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409967 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409970 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409973 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409976 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409978 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409981 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409984 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409987 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409989 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.416617 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409992 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409995 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.409997 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410000 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410002 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410005 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410008 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410010 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410013 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410016 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410019 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410021 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410024 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410026 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410029 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410032 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410034 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410037 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410039 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410042 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.417056 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410044 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410047 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410049 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410053 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410055 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410058 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410061 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410064 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410066 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410069 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410072 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410074 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410077 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410080 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410082 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410085 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410087 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410090 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410092 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.417515 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410095 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410097 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410100 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410102 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410106 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410108 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410111 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410113 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410116 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410119 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410121 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410123 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410126 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410128 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410131 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410133 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410136 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410138 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410141 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.418082 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410143 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410146 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410148 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410151 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410153 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410156 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410159 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410161 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410164 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410166 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410169 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410171 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410174 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410177 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410180 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410183 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410185 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410188 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410191 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410194 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.418541 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410196 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410199 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410201 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410204 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410206 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410209 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410211 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410214 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410217 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410219 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410222 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410224 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410227 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410229 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410232 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410234 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.419050 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.410257 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410295 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410299 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410302 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410305 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410308 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410311 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410313 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410316 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410320 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410324 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410327 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410330 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410333 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410335 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410338 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410341 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410344 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410346 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410349 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:07.419420 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410351 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410354 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410356 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410359 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410361 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410364 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410366 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410369 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410371 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410374 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410376 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410379 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410382 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410384 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410387 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410390 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410392 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410395 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410398 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410400 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:07.419921 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410403 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410405 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410408 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410411 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410413 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410416 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410418 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410421 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410424 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410427 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410429 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410432 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410435 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410437 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410440 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410442 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410447 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410449 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410452 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:07.420431 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410454 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410458 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410461 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410463 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410466 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410468 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410471 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410473 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410476 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410479 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410481 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410484 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410486 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410489 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410491 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410494 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410496 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410499 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410501 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:07.420967 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410504 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410506 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410509 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410512 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410515 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410517 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410520 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410522 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410525 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410528 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410532 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410535 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410538 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410540 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410545 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410549 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410552 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410555 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410557 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410560 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:07.421786 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410564 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410566 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410569 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410571 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410574 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410577 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410579 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410582 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:07.410586 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.410653 2569 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.413462 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.414551 2569 server.go:1014] "Starting client certificate rotation" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.414652 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.415486 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.416894 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:07.422540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.418723 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:07.428888 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.428860 2569 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:00:07.433499 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.433480 2569 log.go:25] "Validated CRI v1 image API" Oct 30 18:00:07.434359 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.434337 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:00:07.437154 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.437133 2569 fs.go:135] Filesystem UUIDs: map[27391037-0853-4ffd-bd6b-d545fd5794e9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae54cb8d-68dd-45e9-85bd-60a90b363e59:/dev/nvme0n1p4] Oct 30 18:00:07.437216 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.437155 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:00:07.442278 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.442166 2569 manager.go:217] Machine: {Timestamp:2026-10-30 18:00:07.440732458 +0000 UTC m=+0.333958842 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106389 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd55e87506f88ffcfc0170640e27f SystemUUID:ec2fd55e-8750-6f88-ffcf-c0170640e27f BootID:bc6edec1-a293-4972-b1e1-2f7f94bbe004 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:8b:86:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:8b:86:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:78:ff:bb:ce:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:00:07.442278 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.442276 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:00:07.442383 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.442354 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:00:07.445031 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445004 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:00:07.445162 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445034 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:00:07.445204 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445166 2569 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:00:07.445204 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445174 2569 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:00:07.445267 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445209 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:07.445267 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.445227 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:07.446335 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.446326 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:00:07.450679 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.450666 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:00:07.450791 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.450782 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:00:07.451625 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451615 2569 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:00:07.451668 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451629 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:00:07.451668 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451639 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:00:07.451668 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451648 2569 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:00:07.451668 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451656 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:00:07.451974 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.451961 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:07.454543 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.454521 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:00:07.461187 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.461172 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:07.462053 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462041 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:00:07.462099 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462071 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:00:07.462225 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462215 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:00:07.462264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462233 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:00:07.462264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462241 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:00:07.462264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462246 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:00:07.462264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462251 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:00:07.462264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462257 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:00:07.462997 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462988 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:00:07.463027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.462998 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:00:07.463027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.463005 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:00:07.463027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.463011 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:00:07.463027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.463019 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:00:07.463027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.463028 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:00:07.464509 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.464499 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:00:07.464509 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.464509 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:00:07.470467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.470446 2569 server.go:1263] "Started kubelet" Oct 30 18:00:07.470561 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.470517 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:00:07.470658 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.470590 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:00:07.470721 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.470685 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:00:07.471252 ip-10-0-143-94 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:00:07.472777 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.472763 2569 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:00:07.473230 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.473213 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:00:07.476905 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.476886 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:00:07.479089 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479070 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:00:07.479089 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479080 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:07.479236 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.479160 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.479830 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479817 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:00:07.479900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479820 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:00:07.479900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479853 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:00:07.479963 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479912 2569 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:00:07.479963 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.479923 2569 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:00:07.480196 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480177 2569 factory.go:55] Registering systemd factory Oct 30 18:00:07.480196 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480198 2569 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:00:07.480447 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480428 2569 factory.go:153] Registering CRI-O factory Oct 30 18:00:07.480447 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480448 2569 factory.go:223] Registration of the crio container factory successfully Oct 30 18:00:07.480546 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480498 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:00:07.480546 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480518 2569 factory.go:103] Registering Raw factory Oct 30 18:00:07.480546 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.480533 2569 manager.go:1199] Started watching for new ooms in manager Oct 30 18:00:07.481570 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.481553 2569 manager.go:319] Starting recovery of all containers Oct 30 18:00:07.492616 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.492573 2569 manager.go:324] Recovery completed Oct 30 18:00:07.494876 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.494855 2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 18:00:07.497754 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.497726 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.501725 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.501704 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.501799 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.501743 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.501799 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.501767 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.502362 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502350 2569 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:00:07.502362 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502361 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:00:07.502448 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502377 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:00:07.502642 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502626 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:00:07.502697 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.502623 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:00:07.502697 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502672 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.502954 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.502933 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:07.503554 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.503537 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.503643 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.503573 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.503643 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.503589 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.503911 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.502740 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-94.ec2.internal.18e36086836c9f7e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-94.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-94.ec2.internal,},FirstTimestamp:2026-10-30 18:00:07.47040755 +0000 UTC m=+0.363633935,LastTimestamp:2026-10-30 18:00:07.47040755 +0000 UTC m=+0.363633935,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-94.ec2.internal,}" Oct 30 18:00:07.504680 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.504668 2569 policy_none.go:50] "Start" Oct 30 18:00:07.504739 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.504686 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:00:07.504739 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.504698 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:00:07.506766 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.506753 2569 policy_none.go:44] "Start" Oct 30 18:00:07.516649 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.516620 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 18:00:07.516649 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.516555 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-94.ec2.internal.18e36086854a8482 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-94.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-94.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-94.ec2.internal,},FirstTimestamp:2026-10-30 18:00:07.50172685 +0000 UTC m=+0.394953241,LastTimestamp:2026-10-30 18:00:07.50172685 +0000 UTC m=+0.394953241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-94.ec2.internal,}" Oct 30 18:00:07.516760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.516744 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:00:07.516802 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.516622 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 18:00:07.528504 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.528372 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kr2dd" Oct 30 18:00:07.536342 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.536326 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kr2dd" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.547209 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.549812 2569 manager.go:341] "Starting Device Plugin manager" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.549841 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.549851 2569 server.go:87] "Starting device plugin registration server" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.550083 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.550095 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.550190 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.550262 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.550270 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.550869 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.550919 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.552558 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-94.ec2.internal.18e36086854b0389 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-94.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-94.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-94.ec2.internal,},FirstTimestamp:2026-10-30 18:00:07.501759369 +0000 UTC m=+0.394985753,LastTimestamp:2026-10-30 18:00:07.501759369 +0000 UTC m=+0.394985753,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-94.ec2.internal,}" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.555322 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.556638 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.556658 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.556692 2569 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:00:07.560098 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.556721 2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:00:07.601286 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.601263 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:07.650240 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.650154 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.656695 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656670 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.656797 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656701 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.656797 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656711 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.656797 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656738 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.656886 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656813 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal"] Oct 30 18:00:07.656921 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.656903 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.658071 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.658056 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.658143 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.658084 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.658143 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.658093 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.660255 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.660244 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.660413 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.660396 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.660480 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.660435 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.661236 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661221 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.661324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661246 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.661324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661255 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.661324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661246 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.661324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661296 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.661324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.661313 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.663378 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.663363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.663452 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.663388 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:07.664134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.664108 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:07.664206 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.664146 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:07.664206 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.664160 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:07.676700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.676686 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.676770 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.676707 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-94.ec2.internal\": node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.680467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.680451 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.680511 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.680475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.680511 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.680493 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.686114 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.686098 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-94.ec2.internal\" not found" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.688115 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.688101 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-94.ec2.internal\" not found" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.732800 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.732767 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.780697 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.780783 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.780783 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.780783 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.780783 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.780783 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.780778 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.833632 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.833591 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.934525 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:07.934470 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:07.991281 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.991255 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:07.993411 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:07.993389 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:08.035058 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.035016 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.135734 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.135690 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.236228 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.236148 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.336700 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.336656 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.414239 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.414213 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:00:08.414716 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.414353 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:08.437540 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.437514 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.507204 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.507131 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:08.537678 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.537649 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.542608 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.542558 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:07 +0000 UTC" deadline="2028-04-24 02:55:53.926662263 +0000 UTC" Oct 30 18:00:08.542608 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.542607 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12992h55m45.384072359s" Oct 30 18:00:08.590929 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.590896 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:08.592798 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.592780 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:08.592905 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.592812 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:08.592905 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.592829 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:08.637920 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.637889 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.659940 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:08.659910 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5baff094b56b89bb07e9432557fee4f.slice/crio-9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0 WatchSource:0}: Error finding container 9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0: Status 404 returned error can't find the container with id 9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0 Oct 30 18:00:08.660137 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:08.660118 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fcd38864cc26968aedfdf83ce5b096.slice/crio-6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493 WatchSource:0}: Error finding container 6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493: Status 404 returned error can't find the container with id 6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493 Oct 30 18:00:08.666528 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.666505 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:00:08.738918 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.738883 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.842956 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:08.842920 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:08.918309 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.918282 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:08.980218 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:08.980185 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:09.003316 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.003275 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:09.005562 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.005548 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:09.029258 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.029194 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:09.030537 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.030520 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:09.453355 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.453278 2569 apiserver.go:52] "Watching apiserver" Oct 30 18:00:09.481076 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.480993 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:09.482958 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.482936 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:00:09.483330 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.483307 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-jk8c2","kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal","openshift-dns/node-resolver-f72ft","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal","openshift-multus/network-metrics-daemon-79lsr","kube-system/global-pull-secret-syncer-jzwlg","openshift-cluster-node-tuning-operator/tuned-rz2lx","openshift-multus/multus-additional-cni-plugins-4wtdk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd","openshift-image-registry/node-ca-rzdgf","openshift-network-diagnostics/network-check-target-2tn9m","openshift-ovn-kubernetes/ovnkube-node-lkmzz","kube-system/konnectivity-agent-dw7mk","openshift-multus/multus-xb89g"] Oct 30 18:00:09.486085 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.486028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.488842 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.488821 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.491049 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.491024 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.493862 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.493842 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.493971 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.493910 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:09.496322 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.496303 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.496429 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.496370 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:09.498643 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.498605 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.501251 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.501228 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.504633 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.504589 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.507207 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.507183 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.508810 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.508787 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:00:09.511171 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.510008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:09.511331 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.511267 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:09.512174 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.512139 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:00:09.512486 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.512471 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:00:09.512793 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.512770 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.512883 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.512856 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:00:09.513015 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.512999 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.515430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.515091 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.517904 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.517883 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:00:09.518113 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.518090 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.519208 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519068 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.519318 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519261 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.519318 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519297 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:00:09.519481 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519394 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.519481 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519408 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:00:09.519584 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.519487 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.522628 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.522608 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.563205 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.563115 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerStarted","Data":"9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0"} Oct 30 18:00:09.564874 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.564837 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" event={"ID":"00fcd38864cc26968aedfdf83ce5b096","Type":"ContainerStarted","Data":"6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493"} Oct 30 18:00:09.567700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.565894 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:09.567700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.565971 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnw47" Oct 30 18:00:09.570557 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.570532 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:09.580977 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.580947 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:00:09.582996 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.582965 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnw47" Oct 30 18:00:09.589691 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.589789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.589789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.589789 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589745 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.589916 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589808 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.589916 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:09.589916 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.589916 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.589932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnd2\" (UniqueName: \"kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g8xv\" (UniqueName: \"kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590068 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590128 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590125 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590149 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590446 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590560 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm68m\" (UniqueName: \"kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qnt\" (UniqueName: \"kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590674 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590724 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590774 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.590898 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xbb\" (UniqueName: \"kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.590996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591176 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591207 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx2r\" (UniqueName: \"kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.591414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591272 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591301 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591360 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqxd\" (UniqueName: \"kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mw8\" (UniqueName: \"kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591508 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.591946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591555 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591666 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591735 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fszc\" (UniqueName: \"kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.592559 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.591941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.614347 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.614318 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:00:09.614606 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.614582 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:00:09.614920 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.614902 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:00:09.615045 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615032 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:00:09.615125 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615115 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:00:09.615185 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615174 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:00:09.615249 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615239 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:00:09.615305 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615264 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:00:09.615360 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615318 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:00:09.615416 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615362 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:00:09.615416 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.615390 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:00:09.617328 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617291 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.617328 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617299 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.617469 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617367 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.617469 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617403 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.617469 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617417 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:09.617469 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617453 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:00:09.617671 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617476 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:00:09.617671 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617536 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.617775 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617700 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:00:09.617775 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617703 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:00:09.617775 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.617724 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:00:09.692775 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xbb\" (UniqueName: \"kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692789 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.692944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692959 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.692966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693028 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx2r\" (UniqueName: \"kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693120 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.693261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqxd\" (UniqueName: \"kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mw8\" (UniqueName: \"kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693587 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693614 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.693972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693731 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693796 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693813 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693848 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fszc\" (UniqueName: \"kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693932 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693989 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694003 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694088 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694108 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.694945 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694345 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694386 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694420 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694444 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694492 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694567 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694650 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694670 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.693913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694843 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694959 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.695848 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694978 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695009 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695036 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.695042 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695076 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnd2\" (UniqueName: \"kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.694927 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8xv\" (UniqueName: \"kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.696769 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.695578 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.695656 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:10.195611191 +0000 UTC m=+3.088837579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695684 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.695702 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:10.195678122 +0000 UTC m=+3.088904511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:09.697555 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm68m\" (UniqueName: \"kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qnt\" (UniqueName: \"kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695959 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696017 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696099 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696148 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698157 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696402 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.695739 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696463 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696466 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696403 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696507 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696517 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696547 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696603 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696654 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.698891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696720 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696800 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696846 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696976 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.696984 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.697063 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.697159 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.698172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.699296 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.699592 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.699324 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.700172 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.700145 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.717356 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.717169 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:09.717356 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.717199 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:09.717356 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.717214 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:09.717356 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:09.717290 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:10.217270205 +0000 UTC m=+3.110496603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:09.719150 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.719079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xbb\" (UniqueName: \"kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.720863 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.720834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8xv\" (UniqueName: \"kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.721129 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.721104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnd2\" (UniqueName: \"kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.721528 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.721504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqxd\" (UniqueName: \"kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:09.724212 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.723857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qnt\" (UniqueName: \"kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.724212 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.724181 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm68m\" (UniqueName: \"kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.726545 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.726496 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mw8\" (UniqueName: \"kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.728301 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.728279 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fszc\" (UniqueName: \"kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.728391 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.728287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx2r\" (UniqueName: \"kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.828122 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.828079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:09.831297 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.831271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:09.843318 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.843282 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dea29c6_0a9e_4f81_8237_f9199715657f.slice/crio-db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4 WatchSource:0}: Error finding container db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4: Status 404 returned error can't find the container with id db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4 Oct 30 18:00:09.843434 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.843340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:09.844191 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.844137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:09.845317 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.845291 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5af711_aeb9_4984_b639_1d6b13db2909.slice/crio-451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c WatchSource:0}: Error finding container 451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c: Status 404 returned error can't find the container with id 451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c Oct 30 18:00:09.848319 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.848219 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:09.855705 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.855640 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe91e54_4f38_43ac_a520_2fc8e083bea9.slice/crio-b51649630c12bebfa944dac8ed83969eb502a637edfeb4dfe14e10203875a86f WatchSource:0}: Error finding container b51649630c12bebfa944dac8ed83969eb502a637edfeb4dfe14e10203875a86f: Status 404 returned error can't find the container with id b51649630c12bebfa944dac8ed83969eb502a637edfeb4dfe14e10203875a86f Oct 30 18:00:09.857355 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.857336 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:09.864117 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.863933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:09.865747 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.865719 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfe69b0_a077_4fd0_9f7e_a513c1b882bf.slice/crio-cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e WatchSource:0}: Error finding container cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e: Status 404 returned error can't find the container with id cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e Oct 30 18:00:09.866798 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.866765 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712baeaa_edb0_45ce_bcd4_9a29c7a5eec0.slice/crio-dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8 WatchSource:0}: Error finding container dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8: Status 404 returned error can't find the container with id dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8 Oct 30 18:00:09.873459 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.873414 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2166c14d_2366_4925_9576_1fccab9df12c.slice/crio-e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35 WatchSource:0}: Error finding container e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35: Status 404 returned error can't find the container with id e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35 Oct 30 18:00:09.874540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.874516 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:09.876132 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.876081 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69fc23e_3928_4ff3_8c03_26c5b5aca388.slice/crio-cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c WatchSource:0}: Error finding container cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c: Status 404 returned error can't find the container with id cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c Oct 30 18:00:09.881052 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:09.881023 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xb89g" Oct 30 18:00:09.892942 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:09.892913 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8385728_e45b_41c9_b2cb_cce86024723b.slice/crio-cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6 WatchSource:0}: Error finding container cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6: Status 404 returned error can't find the container with id cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6 Oct 30 18:00:10.202293 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.202249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:10.202513 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.202305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:10.202513 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.202416 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:10.202513 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.202438 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:10.202513 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.202488 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:11.202472154 +0000 UTC m=+4.095698530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:10.202513 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.202509 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:11.202497898 +0000 UTC m=+4.095724276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:10.303518 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.303483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:10.303716 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.303625 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:10.303716 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.303669 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:10.303716 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.303680 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:10.303890 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.303738 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:11.30372044 +0000 UTC m=+4.196946827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:10.557849 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.557563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:10.558558 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:10.557998 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:10.574112 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.574071 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dw7mk" event={"ID":"b440ae4e-411c-48fd-a8ed-45596e6b4147","Type":"ContainerStarted","Data":"52511988ee22a11f593e0e82b46eaaa17fbf57b62ce424ece5e4b274a34adc09"} Oct 30 18:00:10.579572 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.579467 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8"} Oct 30 18:00:10.582314 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.582244 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" event={"ID":"cfe91e54-4f38-43ac-a520-2fc8e083bea9","Type":"ContainerStarted","Data":"b51649630c12bebfa944dac8ed83969eb502a637edfeb4dfe14e10203875a86f"} Oct 30 18:00:10.585795 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.585738 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:09 +0000 UTC" deadline="2028-06-19 23:37:20.207837649 +0000 UTC" Oct 30 18:00:10.585795 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.585796 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14357h37m9.62204547s" Oct 30 18:00:10.591311 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.591282 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzdgf" event={"ID":"7d5af711-aeb9-4984-b639-1d6b13db2909","Type":"ContainerStarted","Data":"451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c"} Oct 30 18:00:10.596017 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.595105 2569 generic.go:356] "Generic (PLEG): container finished" podID="e5baff094b56b89bb07e9432557fee4f" containerID="47fa4420b4cc6b164de9d46b9c8e441f6f0583d1f1f94f73b0fdd59452c31fde" exitCode=0 Oct 30 18:00:10.596017 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.595193 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerDied","Data":"47fa4420b4cc6b164de9d46b9c8e441f6f0583d1f1f94f73b0fdd59452c31fde"} Oct 30 18:00:10.603692 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.603663 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" event={"ID":"00fcd38864cc26968aedfdf83ce5b096","Type":"ContainerStarted","Data":"757e506de47e59c0394b0ac21822f457395771318431becd6041d500610f1106"} Oct 30 18:00:10.621630 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.621593 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xb89g" event={"ID":"d8385728-e45b-41c9-b2cb-cce86024723b","Type":"ContainerStarted","Data":"cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6"} Oct 30 18:00:10.624761 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.624688 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jk8c2" event={"ID":"f69fc23e-3928-4ff3-8c03-26c5b5aca388","Type":"ContainerStarted","Data":"cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c"} Oct 30 18:00:10.649103 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.648997 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" event={"ID":"2166c14d-2366-4925-9576-1fccab9df12c","Type":"ContainerStarted","Data":"e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35"} Oct 30 18:00:10.650956 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.650887 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f72ft" event={"ID":"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf","Type":"ContainerStarted","Data":"cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e"} Oct 30 18:00:10.653462 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:10.653398 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerStarted","Data":"db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4"} Oct 30 18:00:11.211182 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.211087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:11.211182 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.211160 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:11.211474 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.211334 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:11.211474 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.211401 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:13.211381371 +0000 UTC m=+6.104607769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:11.211875 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.211853 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:11.211975 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.211918 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:13.211901536 +0000 UTC m=+6.105127925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:11.312825 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.312122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:11.312825 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.312309 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:11.312825 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.312335 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:11.312825 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.312349 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:11.312825 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.312412 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:13.312393022 +0000 UTC m=+6.205619423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:11.557894 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.557819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:11.557894 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.557864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:11.558339 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.557963 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:11.558447 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:11.558420 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:11.586940 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.586837 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:09 +0000 UTC" deadline="2028-08-02 22:13:45.019760078 +0000 UTC" Oct 30 18:00:11.586940 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.586894 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15412h13m33.432870458s" Oct 30 18:00:11.668625 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.668546 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerStarted","Data":"edb00c8462801837e811135b920422d1d7dc112717892e6abd010a87a822d2e2"} Oct 30 18:00:11.714413 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.713416 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" podStartSLOduration=2.71339515 podStartE2EDuration="2.71339515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:10.679877458 +0000 UTC m=+3.573103855" watchObservedRunningTime="2026-10-30 18:00:11.71339515 +0000 UTC m=+4.606621547" Oct 30 18:00:11.714413 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:11.713579 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" podStartSLOduration=2.713571558 podStartE2EDuration="2.713571558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:11.708799324 +0000 UTC m=+4.602025746" watchObservedRunningTime="2026-10-30 18:00:11.713571558 +0000 UTC m=+4.606797958" Oct 30 18:00:12.557687 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:12.557085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:12.557687 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:12.557240 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:12.681825 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:12.680883 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="e2739eee1dd8812468dcb479e3d306419bb4efe3635912334522b08b6da33c2c" exitCode=0 Oct 30 18:00:12.681825 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:12.681065 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"e2739eee1dd8812468dcb479e3d306419bb4efe3635912334522b08b6da33c2c"} Oct 30 18:00:13.227084 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:13.227033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:13.227255 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:13.227096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:13.227255 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.227235 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:13.227388 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.227294 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:17.22727708 +0000 UTC m=+10.120503469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:13.227736 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.227715 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:13.227833 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.227763 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:17.227753469 +0000 UTC m=+10.120979853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:13.328173 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:13.328135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:13.328369 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.328332 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:13.328369 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.328351 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:13.328369 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.328362 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:13.328527 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.328416 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:17.328397654 +0000 UTC m=+10.221624031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:13.558900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:13.558829 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:13.559060 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.558958 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:13.559244 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:13.559228 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:13.559314 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:13.559299 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:14.557211 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:14.557176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:14.557760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:14.557349 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:14.695192 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:14.695155 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dw7mk" event={"ID":"b440ae4e-411c-48fd-a8ed-45596e6b4147","Type":"ContainerStarted","Data":"381859a5ca1c020104bfc0a5a70ba36443de29e22a4a23185f57dd16d1c7bd49"} Oct 30 18:00:14.734447 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:14.734094 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dw7mk" podStartSLOduration=2.816944488 podStartE2EDuration="6.734081324s" totalImagesPullingTime="3.917136836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:14.73372306 +0000 UTC m=+7.626949454" watchObservedRunningTime="2026-10-30 18:00:14.734081324 +0000 UTC m=+7.627307717" Oct 30 18:00:15.294459 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:15.294089 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:15.294717 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:15.294611 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:15.560054 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:15.559987 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:15.560517 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:15.560116 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:15.560560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:15.560517 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:15.560645 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:15.560623 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:15.702488 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:15.702246 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzdgf" event={"ID":"7d5af711-aeb9-4984-b639-1d6b13db2909","Type":"ContainerStarted","Data":"d1b038aa79e2d23daaced3938f947566cfbf3fb538c931c2d61aae879e044ffb"} Oct 30 18:00:16.557777 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:16.557237 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:16.557777 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:16.557388 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:16.706904 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:16.706872 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" event={"ID":"2166c14d-2366-4925-9576-1fccab9df12c","Type":"ContainerStarted","Data":"620d62f797c910ac25ef30988e968696851fa03be7b1e4c476599561c34d49f5"} Oct 30 18:00:16.707261 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:16.706964 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:17.224615 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.224568 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:00:17.266717 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.266245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:17.266717 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.266287 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:17.266717 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.266379 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:17.266717 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.266434 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:25.266416914 +0000 UTC m=+18.159643285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:17.267027 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.266767 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:17.267027 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.266827 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:25.266811321 +0000 UTC m=+18.160037695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:17.367155 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.367115 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:17.367369 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.367350 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:17.367444 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.367379 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:17.367444 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.367394 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:17.367542 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.367455 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:25.367434953 +0000 UTC m=+18.260661343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:17.555159 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.555044 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:00:17.224588918Z","UUID":"8726f55c-0897-49eb-af99-0ee181bd4d78","Handler":null,"Name":"","Endpoint":""} Oct 30 18:00:17.559795 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.559755 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:00:17.559795 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.559787 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:00:17.560003 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.559847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:17.560003 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.559955 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:17.560352 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.560328 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:17.561518 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:17.561482 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:17.711868 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.711785 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" event={"ID":"2166c14d-2366-4925-9576-1fccab9df12c","Type":"ContainerStarted","Data":"89ff1e409229ec1394b19d1c825095e9e0bfe977c10b1fc3c070b859661eeb51"} Oct 30 18:00:17.716052 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.716019 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="d262cc914c6d76f7a8933e6bbc7b7c8b765e6e2c96cbea7a16b46394cfd230c2" exitCode=0 Oct 30 18:00:17.716204 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.716067 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"d262cc914c6d76f7a8933e6bbc7b7c8b765e6e2c96cbea7a16b46394cfd230c2"} Oct 30 18:00:17.759565 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:17.759516 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rzdgf" podStartSLOduration=4.62333555 podStartE2EDuration="9.759501459s" totalImagesPullingTime="5.136165909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:15.740493824 +0000 UTC m=+8.633720219" watchObservedRunningTime="2026-10-30 18:00:17.759501459 +0000 UTC m=+10.652727859" Oct 30 18:00:18.557900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.557870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:18.558005 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:18.557986 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:18.720244 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.720205 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xb89g" event={"ID":"d8385728-e45b-41c9-b2cb-cce86024723b","Type":"ContainerStarted","Data":"1425fbd3ed1b8a88370633e2c53b4f789e0a23632e436d8cc8101607acf88aa3"} Oct 30 18:00:18.724005 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.723971 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" event={"ID":"2166c14d-2366-4925-9576-1fccab9df12c","Type":"ContainerStarted","Data":"06ed1106bbba43138c1972fc22b37ff434dd237127f08eeca31a33693e5b0c25"} Oct 30 18:00:18.726084 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.726048 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f72ft" event={"ID":"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf","Type":"ContainerStarted","Data":"578b335cd4269d51dbcd59e3b1287bd3d645959405f6a31d44ef60f1895cd560"} Oct 30 18:00:18.729308 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.729280 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="ad322a50215855ac5e212ce1db47469d7bd1d94364a1226806f6855b7bb83af5" exitCode=0 Oct 30 18:00:18.729426 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.729319 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"ad322a50215855ac5e212ce1db47469d7bd1d94364a1226806f6855b7bb83af5"} Oct 30 18:00:18.758541 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.758462 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xb89g" podStartSLOduration=2.728613841 podStartE2EDuration="10.758444662s" totalImagesPullingTime="8.029830821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:18.758165014 +0000 UTC m=+11.651391428" watchObservedRunningTime="2026-10-30 18:00:18.758444662 +0000 UTC m=+11.651671047" Oct 30 18:00:18.795318 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.795211 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-f72ft" podStartSLOduration=2.421168715 podStartE2EDuration="10.795197246s" totalImagesPullingTime="8.374028531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:18.79432475 +0000 UTC m=+11.687551143" watchObservedRunningTime="2026-10-30 18:00:18.795197246 +0000 UTC m=+11.688423638" Oct 30 18:00:18.835649 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:18.835576 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" podStartSLOduration=2.736699078 podStartE2EDuration="10.835561725s" totalImagesPullingTime="8.098862647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:18.834970889 +0000 UTC m=+11.728197281" watchObservedRunningTime="2026-10-30 18:00:18.835561725 +0000 UTC m=+11.728788118" Oct 30 18:00:19.557238 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.557198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:19.557410 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:19.557388 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:19.557736 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.557717 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:19.557837 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:19.557819 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:19.737570 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.737484 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jk8c2" event={"ID":"f69fc23e-3928-4ff3-8c03-26c5b5aca388","Type":"ContainerStarted","Data":"ffa3fcc2db4c1998df596f871732aa9766c9df1da914667d8ea11d37cc1189b7"} Oct 30 18:00:19.740972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.740941 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="ee01ec95acfeebf9566d72903fd1f500936e0d5ff92f75366f712d246274a3d6" exitCode=0 Oct 30 18:00:19.741086 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.741045 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"ee01ec95acfeebf9566d72903fd1f500936e0d5ff92f75366f712d246274a3d6"} Oct 30 18:00:19.774325 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:19.774273 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jk8c2" podStartSLOduration=3.706778634 podStartE2EDuration="11.774258772s" totalImagesPullingTime="8.067480138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:19.773615868 +0000 UTC m=+12.666842258" watchObservedRunningTime="2026-10-30 18:00:19.774258772 +0000 UTC m=+12.667485185" Oct 30 18:00:20.557748 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:20.557469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:20.557918 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:20.557835 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:20.746231 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:20.745808 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" event={"ID":"cfe91e54-4f38-43ac-a520-2fc8e083bea9","Type":"ContainerStarted","Data":"3dbd5767bc770c8e1e73f211e9728f638c4ab18a18c60af8ccb68fb4a85b4e3b"} Oct 30 18:00:20.785934 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:20.785874 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" podStartSLOduration=2.471801154 podStartE2EDuration="12.785856686s" totalImagesPullingTime="10.314055532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:20.785131878 +0000 UTC m=+13.678358271" watchObservedRunningTime="2026-10-30 18:00:20.785856686 +0000 UTC m=+13.679083081" Oct 30 18:00:21.560990 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:21.560962 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:21.561165 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:21.561119 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:21.561165 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:21.561158 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:21.561276 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:21.561252 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:22.556935 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:22.556905 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:22.557897 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:22.557036 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:22.754131 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:22.754092 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="8b35a5e5b804874accdbf62bcd8bf370ccf2b8b0a803fef79da01547a26acf36" exitCode=0 Oct 30 18:00:22.754279 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:22.754171 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"8b35a5e5b804874accdbf62bcd8bf370ccf2b8b0a803fef79da01547a26acf36"} Oct 30 18:00:23.559940 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:23.559908 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:23.559940 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:23.559930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:23.560445 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:23.560031 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:23.560445 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:23.560166 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:23.759773 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:23.759740 2569 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="3f94680a9db51268516226ae389c0e7e926987c663d3d90079c4eb9bfbc627d0" exitCode=0 Oct 30 18:00:23.759927 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:23.759805 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerDied","Data":"3f94680a9db51268516226ae389c0e7e926987c663d3d90079c4eb9bfbc627d0"} Oct 30 18:00:24.557900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.557698 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:24.558136 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:24.557995 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:24.766087 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.766057 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" event={"ID":"7dea29c6-0a9e-4f81-8237-f9199715657f","Type":"ContainerStarted","Data":"49d36cfc6c46e2ce4c8a91bf4a68d415bdc1fea123878df2ddb685285676d0b9"} Oct 30 18:00:24.768175 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.768144 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"998894acf9f5260cfb93da5c7e16d205dc62689a80d001d8d05cb602882a3271"} Oct 30 18:00:24.768266 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.768212 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"4391848cc6ea1f611d735eb5e5ff15f8cfbd8021fca1e4d5f040e58b54967ace"} Oct 30 18:00:24.768266 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.768242 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"9ea3ddcaf1edfe7c9251eb8adef3fcb6861855c18b5307190fb0a652cce2570c"} Oct 30 18:00:24.832636 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:24.832573 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wtdk" podStartSLOduration=7.782167604 podStartE2EDuration="17.832560328s" totalImagesPullingTime="10.050392724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:24.832053743 +0000 UTC m=+17.725280137" watchObservedRunningTime="2026-10-30 18:00:24.832560328 +0000 UTC m=+17.725786721" Oct 30 18:00:25.326786 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.326749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:25.326946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.326804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:25.326946 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.326892 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:25.326946 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.326936 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:25.327083 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.326955 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret podName:dca34023-3797-4998-9674-ee14daf13450 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.32694151 +0000 UTC m=+34.220167880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret") pod "global-pull-secret-syncer-jzwlg" (UID: "dca34023-3797-4998-9674-ee14daf13450") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:25.327083 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.326988 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.326973087 +0000 UTC m=+34.220199481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:25.427224 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.427189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:25.427395 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.427360 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:25.427395 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.427384 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:25.427395 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.427394 2569 projected.go:196] Error preparing data for projected volume kube-api-access-mw28h for pod openshift-network-diagnostics/network-check-target-2tn9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:25.427492 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.427441 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h podName:a7517a84-07c0-4210-9303-ac839cdd188b nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.427429266 +0000 UTC m=+34.320655638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mw28h" (UniqueName: "kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h") pod "network-check-target-2tn9m" (UID: "a7517a84-07c0-4210-9303-ac839cdd188b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:25.557936 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.557894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:25.558114 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.558038 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:25.558114 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.558074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:25.558223 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:25.558144 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:25.773076 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.772846 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"ddd11f2aeca70ac9b4f489f786b19635f4bd77a6ca1a2cd5c15c2073e7907b29"} Oct 30 18:00:25.773076 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.773069 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"08673affce5dfe62b2a0d3c5e7dc3637f913495930d1afa1d3dee350e8f9e691"} Oct 30 18:00:25.773444 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:25.773103 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"8ff7f1f8f50d8d96103fe46cd573afa69e38f469fde7c528c2b65d6d8ab082f6"} Oct 30 18:00:26.557755 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:26.557718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:26.557908 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:26.557831 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:27.557670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:27.557636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:27.558084 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:27.557735 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:27.558084 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:27.557790 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:27.558084 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:27.557903 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:27.780336 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:27.780295 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"5af5424f819351104ebf63e35c45e947958a863ef1cb823aa04b2aabeca092f4"} Oct 30 18:00:28.557312 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:28.557278 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:28.557460 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:28.557386 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:29.559998 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.559966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:29.559998 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.560002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:29.560838 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:29.560092 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:29.560838 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:29.560209 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:29.787520 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.787483 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" event={"ID":"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0","Type":"ContainerStarted","Data":"984d61e265626cd39a9b735a98cf17bf3b230863dd7d826761828933b3d2c74b"} Oct 30 18:00:29.787825 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.787759 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:29.803048 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.803016 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:29.869234 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:29.869138 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" podStartSLOduration=8.169719483 podStartE2EDuration="22.869123904s" totalImagesPullingTime="14.699404421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:29.869103054 +0000 UTC m=+22.762329448" watchObservedRunningTime="2026-10-30 18:00:29.869123904 +0000 UTC m=+22.762350289" Oct 30 18:00:30.557350 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:30.557316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:30.557518 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:30.557430 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:30.790457 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:30.790428 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:30.790950 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:30.790501 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:30.805467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:30.805444 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:31.170159 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.170121 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jzwlg"] Oct 30 18:00:31.170350 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.170258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:31.170421 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:31.170366 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:31.174880 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.174851 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tn9m"] Oct 30 18:00:31.175019 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.174957 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:31.175068 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:31.175034 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:31.178032 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.178007 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79lsr"] Oct 30 18:00:31.178140 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:31.178088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:31.178208 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:31.178189 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:32.557281 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:32.557249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:32.557725 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:32.557353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:32.557725 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:32.557369 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:32.557725 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:32.557422 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:32.557725 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:32.557439 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:32.557725 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:32.557532 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:34.289302 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.289092 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:34.289784 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.289427 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:34.289784 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.289661 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:34.557517 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.557445 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:34.557517 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.557481 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:34.557711 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:34.557555 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:34.557711 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:34.557574 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:34.557711 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:34.557639 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:34.557810 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:34.557719 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:36.557756 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:36.557715 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:36.558202 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:36.557722 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:36.558202 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:36.557822 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tn9m" podUID="a7517a84-07c0-4210-9303-ac839cdd188b" Oct 30 18:00:36.558202 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:36.557735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:36.558202 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:36.557899 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-79lsr" podUID="655bf5c4-4149-4e0e-b03b-10a41946b0f1" Oct 30 18:00:36.558202 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:36.557966 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jzwlg" podUID="dca34023-3797-4998-9674-ee14daf13450" Oct 30 18:00:37.344095 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.344069 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeReady" Oct 30 18:00:37.344219 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.344208 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:00:37.389132 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.389059 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ztx4c"] Oct 30 18:00:37.394148 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.394119 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6m5w"] Oct 30 18:00:37.394301 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.394211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:37.396282 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.396263 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzr8f"] Oct 30 18:00:37.396391 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.396373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:37.399385 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.399368 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfmlk"] Oct 30 18:00:37.399514 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.399495 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.403793 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.403420 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx"] Oct 30 18:00:37.404315 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.404290 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:37.409145 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.409121 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7765667bbf-4zpgd"] Oct 30 18:00:37.409821 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.409792 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.412460 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.412441 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.413797 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.413772 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w"] Oct 30 18:00:37.413944 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.413911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.415959 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.415938 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.418045 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.417223 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:00:37.418045 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.417866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.418451 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.418435 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 18:00:37.420006 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.419985 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l"] Oct 30 18:00:37.420119 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.420099 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.423862 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.423838 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6"] Oct 30 18:00:37.424025 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.424008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.429760 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.429736 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ts48c"] Oct 30 18:00:37.429876 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.429856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.432074 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.432054 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd"] Oct 30 18:00:37.432561 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.432542 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.436693 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.436671 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:00:37.437740 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.437720 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:37.439767 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.439749 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj"] Oct 30 18:00:37.439891 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.439875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.446050 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.446027 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pllqc"] Oct 30 18:00:37.449543 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.448891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:37.450822 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.450804 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fnrlr"] Oct 30 18:00:37.450987 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.450965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.456932 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.456910 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dq8ks"] Oct 30 18:00:37.460689 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.460667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.464273 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464254 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ztx4c"] Oct 30 18:00:37.464445 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464423 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfmlk"] Oct 30 18:00:37.464445 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464461 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464491 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzr8f"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464522 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464547 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464572 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464619 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464653 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7765667bbf-4zpgd"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464677 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464707 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6m5w"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464737 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464760 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dq8ks"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464787 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnrlr"] Oct 30 18:00:37.465298 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.464893 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.466858 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.466840 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:00:37.472665 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.472643 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pllqc"] Oct 30 18:00:37.475781 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.475761 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ts48c"] Oct 30 18:00:37.510683 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510656 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.510826 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510807 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510827 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510846 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510883 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510896 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510914 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510940 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510947 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511026 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.510991 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511063 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511102 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511155 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511194 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511352 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511364 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 18:00:37.511430 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511413 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 18:00:37.511755 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511435 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511755 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511439 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511864 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511839 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.511864 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511850 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511957 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511874 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511957 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511901 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511957 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511916 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:00:37.511957 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.511955 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.512099 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.512020 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 18:00:37.512099 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.512022 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.512206 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.512193 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:37.515143 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbksp\" (UniqueName: \"kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.515245 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515245 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sqc\" (UniqueName: \"kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515335 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515335 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.515409 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k227\" (UniqueName: \"kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.515409 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.515476 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.515523 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515587 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515565 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.515705 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4njc\" (UniqueName: \"kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.515761 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.515814 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515869 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.515869 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.515946 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.515979 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5cm\" (UniqueName: \"kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm\") pod \"kas-connection-checker-685895c7c9-ztx4c\" (UID: \"5084064f-9031-4a3c-b74f-46abf27982a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:37.515979 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.516072 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.515999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78ql\" (UniqueName: \"kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.516072 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hhd\" (UniqueName: \"kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd\") pod \"kas-connection-checker-685895c7c9-l6m5w\" (UID: \"3b070856-f46c-4ebb-965c-d8aaacb18baf\") " pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:37.516164 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdb7l\" (UniqueName: \"kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l\") pod \"network-check-source-6f78d59f99-7ncwj\" (UID: \"0239d118-e691-4ac3-9e13-e1ced388a733\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:37.516164 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.516257 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.516257 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bht72\" (UniqueName: \"kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.516257 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.516370 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.516370 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.516472 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516411 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.516472 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.516567 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516496 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82fn6\" (UniqueName: \"kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6\") pod \"volume-data-source-validator-58c9f5f46d-ds2qd\" (UID: \"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:37.516567 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.516700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.516700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516645 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.516700 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.516852 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkwc\" (UniqueName: \"kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.516852 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516772 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.516950 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.516950 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516905 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqf6r\" (UniqueName: \"kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r\") pod \"kas-connection-checker-685895c7c9-lfmlk\" (UID: \"47a7c100-1107-4314-aa33-526e92b42c1f\") " pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:37.516950 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.517044 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.516980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.517088 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.517127 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.517212 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.517251 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.517251 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.517351 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.517351 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517288 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.517447 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.517345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.523771 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523751 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:00:37.523926 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523837 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 18:00:37.523926 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523875 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:00:37.523926 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523911 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 18:00:37.524136 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523959 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:00:37.524136 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.523986 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:00:37.524136 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524011 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 18:00:37.524136 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524098 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 18:00:37.524396 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524187 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 18:00:37.524396 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524197 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 18:00:37.524396 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524293 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 18:00:37.524782 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524763 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6wcdg\"" Oct 30 18:00:37.524873 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524804 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 18:00:37.524873 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524849 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:00:37.524873 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524854 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 18:00:37.525000 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524894 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:00:37.525000 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524922 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8sxnb\"" Oct 30 18:00:37.525000 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524950 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-shsdq\"" Oct 30 18:00:37.525000 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524952 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qhhbf\"" Oct 30 18:00:37.525000 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.524984 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v5bj5\"" Oct 30 18:00:37.525141 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.525006 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wvmzn\"" Oct 30 18:00:37.534359 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.534343 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg98\"" Oct 30 18:00:37.534983 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.534968 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 18:00:37.535064 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.534992 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kkf44\"" Oct 30 18:00:37.535064 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.535008 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kr5jk\"" Oct 30 18:00:37.535208 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.535195 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nsbpz\"" Oct 30 18:00:37.535280 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.535262 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cbhh9\"" Oct 30 18:00:37.535479 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.535468 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:00:37.536499 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.536480 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:00:37.536840 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.536807 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:00:37.536931 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.536918 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 18:00:37.618255 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.618255 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4njc\" (UniqueName: \"kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5cm\" (UniqueName: \"kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm\") pod \"kas-connection-checker-685895c7c9-ztx4c\" (UID: \"5084064f-9031-4a3c-b74f-46abf27982a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78ql\" (UniqueName: \"kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.618712 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618673 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hhd\" (UniqueName: \"kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd\") pod \"kas-connection-checker-685895c7c9-l6m5w\" (UID: \"3b070856-f46c-4ebb-965c-d8aaacb18baf\") " pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618840 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdb7l\" (UniqueName: \"kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l\") pod \"network-check-source-6f78d59f99-7ncwj\" (UID: \"0239d118-e691-4ac3-9e13-e1ced388a733\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.618924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619233 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.619287 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619259 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619324 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619380 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.119363264 +0000 UTC m=+31.012589653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bht72\" (UniqueName: \"kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.619647 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619629 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619674 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls podName:2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.119661085 +0000 UTC m=+31.012887489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nmrh6" (UID: "2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd") : secret "samples-operator-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fn6\" (UniqueName: \"kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6\") pod \"volume-data-source-validator-58c9f5f46d-ds2qd\" (UID: \"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619740 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619753 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f75c69b8c-gmvng: secret "image-registry-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619790 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls podName:3f409e7d-56d0-43ed-ac57-5724283f1a57 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.119774776 +0000 UTC m=+31.013001173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls") pod "image-registry-6f75c69b8c-gmvng" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57") : secret "image-registry-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619847 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkwc\" (UniqueName: \"kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.619980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxdr\" (UniqueName: \"kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.619991 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:37.620134 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.620006 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bf557f7d-4tfww: secret "image-registry-tls" not found Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.620059 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls podName:8528ac59-25e8-435f-9d93-599e19af51c5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.120042845 +0000 UTC m=+31.013269236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls") pod "image-registry-85bf557f7d-4tfww" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5") : secret "image-registry-tls" not found Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqf6r\" (UniqueName: \"kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r\") pod \"kas-connection-checker-685895c7c9-lfmlk\" (UID: \"47a7c100-1107-4314-aa33-526e92b42c1f\") " pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620221 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.620925 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbksp\" (UniqueName: \"kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620438 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sqc\" (UniqueName: \"kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.620562 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.120546236 +0000 UTC m=+31.013772634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : configmap references non-existent config key: service-ca.crt Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k227\" (UniqueName: \"kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq2b\" (UniqueName: \"kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.620828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.621667 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.621166 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:00:37.622382 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.621212 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.121196814 +0000 UTC m=+31.014423204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : secret "router-metrics-certs-default" not found Oct 30 18:00:37.622382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.621981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.622382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.622287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.622575 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.622552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.623877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624186 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624515 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624537 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.624670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.624978 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.625033 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.624980 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.625500 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.625475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.626036 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.626011 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.626445 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.626423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.626614 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.626576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.626869 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.626854 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.627069 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.627046 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.643235 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.643205 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4njc\" (UniqueName: \"kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:37.645699 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.645364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.645800 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.645759 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdb7l\" (UniqueName: \"kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l\") pod \"network-check-source-6f78d59f99-7ncwj\" (UID: \"0239d118-e691-4ac3-9e13-e1ced388a733\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:37.646567 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.646534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fn6\" (UniqueName: \"kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6\") pod \"volume-data-source-validator-58c9f5f46d-ds2qd\" (UID: \"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:37.646670 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.646546 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.647843 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.647797 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqf6r\" (UniqueName: \"kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r\") pod \"kas-connection-checker-685895c7c9-lfmlk\" (UID: \"47a7c100-1107-4314-aa33-526e92b42c1f\") " pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:37.648415 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.648264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:37.648415 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.648367 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5cm\" (UniqueName: \"kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm\") pod \"kas-connection-checker-685895c7c9-ztx4c\" (UID: \"5084064f-9031-4a3c-b74f-46abf27982a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:37.650283 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.650259 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78ql\" (UniqueName: \"kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.650467 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.650434 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bht72\" (UniqueName: \"kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.650995 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.650970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:37.651085 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.651052 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hhd\" (UniqueName: \"kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd\") pod \"kas-connection-checker-685895c7c9-l6m5w\" (UID: \"3b070856-f46c-4ebb-965c-d8aaacb18baf\") " pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:37.651173 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.651150 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbksp\" (UniqueName: \"kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:37.651985 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.651963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sqc\" (UniqueName: \"kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.652085 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.651964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k227\" (UniqueName: \"kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:37.653239 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.653204 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkwc\" (UniqueName: \"kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.709365 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.709333 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:37.718698 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.718679 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:37.722106 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.722223 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxdr\" (UniqueName: \"kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.722731 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.722832 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.722179 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:37.722901 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.722894 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.222874276 +0000 UTC m=+31.116100667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:37.722967 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:37.723022 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.723022 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.722995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.723123 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.723114 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:00:37.723169 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.723152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq2b\" (UniqueName: \"kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.723169 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:37.723161 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls podName:3208b565-97ac-4ed9-9d36-a289544c5fd1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.223145283 +0000 UTC m=+31.116371671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls") pod "dns-default-fnrlr" (UID: "3208b565-97ac-4ed9-9d36-a289544c5fd1") : secret "dns-default-metrics-tls" not found Oct 30 18:00:37.723253 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.723202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.723305 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.723249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.723420 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.723404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.723522 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.723504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.725936 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.725915 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.729703 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.729684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:37.734892 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.734866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:37.744064 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.744038 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxdr\" (UniqueName: \"kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.744794 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.744767 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq2b\" (UniqueName: \"kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:37.764470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.764447 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:37.797512 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.797479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:37.848935 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.848880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:37.884300 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.884165 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:37.894510 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.893569 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:37.956470 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.956435 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:37.956742 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:37.956715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dw7mk_b440ae4e-411c-48fd-a8ed-45596e6b4147/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:38.035424 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.035365 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l6m5w"] Oct 30 18:00:38.037588 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.037560 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ztx4c"] Oct 30 18:00:38.046372 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.046347 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5084064f_9031_4a3c_b74f_46abf27982a3.slice/crio-cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8 WatchSource:0}: Error finding container cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8: Status 404 returned error can't find the container with id cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8 Oct 30 18:00:38.084215 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.084067 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kzr8f"] Oct 30 18:00:38.090713 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.090687 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lfmlk"] Oct 30 18:00:38.095810 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.095787 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w"] Oct 30 18:00:38.100058 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.100027 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891a4e9b_107c_45f6_bef6_d825fe05e079.slice/crio-bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b WatchSource:0}: Error finding container bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b: Status 404 returned error can't find the container with id bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b Oct 30 18:00:38.117294 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.117268 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dw7mk_b440ae4e-411c-48fd-a8ed-45596e6b4147/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:38.132780 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.132747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:38.132900 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.132860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:38.132969 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.132904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:38.132969 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.132929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:38.132969 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.132958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:38.133122 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.133031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:38.133169 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133160 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:00:38.133236 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133220 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133201192 +0000 UTC m=+32.026427583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : secret "router-metrics-certs-default" not found Oct 30 18:00:38.133312 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133299 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133288987 +0000 UTC m=+32.026515377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : configmap references non-existent config key: service-ca.crt Oct 30 18:00:38.133376 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133363 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:38.133423 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133395 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133385098 +0000 UTC m=+32.026611483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:38.133481 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133452 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:38.133481 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133462 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f75c69b8c-gmvng: secret "image-registry-tls" not found Oct 30 18:00:38.133572 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133491 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls podName:3f409e7d-56d0-43ed-ac57-5724283f1a57 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133481955 +0000 UTC m=+32.026708340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls") pod "image-registry-6f75c69b8c-gmvng" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57") : secret "image-registry-tls" not found Oct 30 18:00:38.133572 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133543 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:00:38.133699 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133573 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls podName:2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133562865 +0000 UTC m=+32.026789252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nmrh6" (UID: "2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd") : secret "samples-operator-tls" not found Oct 30 18:00:38.133699 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133684 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:38.133699 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133697 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bf557f7d-4tfww: secret "image-registry-tls" not found Oct 30 18:00:38.133843 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.133747 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls podName:8528ac59-25e8-435f-9d93-599e19af51c5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.133735883 +0000 UTC m=+32.026962270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls") pod "image-registry-85bf557f7d-4tfww" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5") : secret "image-registry-tls" not found Oct 30 18:00:38.142448 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.140663 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l"] Oct 30 18:00:38.146258 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.146200 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e82c61_c992_432e_b1ab_dffdaadccf70.slice/crio-bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59 WatchSource:0}: Error finding container bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59: Status 404 returned error can't find the container with id bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59 Oct 30 18:00:38.166419 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.166385 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ts48c"] Oct 30 18:00:38.178689 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.178661 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd861baa6_45e0_44b6_9336_1ed846e00b49.slice/crio-a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c WatchSource:0}: Error finding container a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c: Status 404 returned error can't find the container with id a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c Oct 30 18:00:38.218004 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.217979 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd"] Oct 30 18:00:38.219305 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.219285 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj"] Oct 30 18:00:38.220396 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.220369 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239d118_e691_4ac3_9e13_e1ced388a733.slice/crio-4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1 WatchSource:0}: Error finding container 4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1: Status 404 returned error can't find the container with id 4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1 Oct 30 18:00:38.221181 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.221162 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cbcc9b_a1ea_4304_9fe7_1805deaf6f03.slice/crio-ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49 WatchSource:0}: Error finding container ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49: Status 404 returned error can't find the container with id ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49 Oct 30 18:00:38.234107 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.234080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:38.234234 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.234215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:38.234338 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.234239 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:00:38.234338 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.234308 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls podName:3208b565-97ac-4ed9-9d36-a289544c5fd1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.234285333 +0000 UTC m=+32.127511721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls") pod "dns-default-fnrlr" (UID: "3208b565-97ac-4ed9-9d36-a289544c5fd1") : secret "dns-default-metrics-tls" not found Oct 30 18:00:38.234430 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.234353 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:38.234430 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:38.234405 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.234389965 +0000 UTC m=+32.127616355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:38.274497 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.274460 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dq8ks"] Oct 30 18:00:38.276354 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:38.276333 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47517669_cee7_4488_827e_b605f1cbfbc0.slice/crio-1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919 WatchSource:0}: Error finding container 1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919: Status 404 returned error can't find the container with id 1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919 Oct 30 18:00:38.557529 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.557446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:38.557529 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.557475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:38.557756 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.557446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:38.573015 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.572994 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:00:38.573134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.573065 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:00:38.573194 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.573180 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:00:38.573324 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.573304 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:00:38.573422 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.573344 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:00:38.827379 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.827290 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" event={"ID":"5084064f-9031-4a3c-b74f-46abf27982a3","Type":"ContainerStarted","Data":"4ed438830d240e287c36bd27f7e15b175a94acc92301a3c8e1165eb33d02cc1f"} Oct 30 18:00:38.827834 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.827382 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" event={"ID":"5084064f-9031-4a3c-b74f-46abf27982a3","Type":"ContainerStarted","Data":"cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8"} Oct 30 18:00:38.830814 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.830782 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" event={"ID":"0239d118-e691-4ac3-9e13-e1ced388a733","Type":"ContainerStarted","Data":"4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1"} Oct 30 18:00:38.832582 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.832534 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" event={"ID":"3b070856-f46c-4ebb-965c-d8aaacb18baf","Type":"ContainerStarted","Data":"e084a0d6c7709b336eba4c828695201425b33dba427c4e7ac1ba3bad3e1578fd"} Oct 30 18:00:38.832582 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.832581 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" event={"ID":"3b070856-f46c-4ebb-965c-d8aaacb18baf","Type":"ContainerStarted","Data":"4cd9755409bd32ba6949dcd62cdd53155f4ed9dfec84003b02a8d10e22ebae58"} Oct 30 18:00:38.833961 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.833927 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dq8ks" event={"ID":"47517669-cee7-4488-827e-b605f1cbfbc0","Type":"ContainerStarted","Data":"1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919"} Oct 30 18:00:38.838060 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.838014 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" event={"ID":"891a4e9b-107c-45f6-bef6-d825fe05e079","Type":"ContainerStarted","Data":"bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b"} Oct 30 18:00:38.839557 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.839510 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" event={"ID":"d861baa6-45e0-44b6-9336-1ed846e00b49","Type":"ContainerStarted","Data":"a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c"} Oct 30 18:00:38.841907 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.841789 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" event={"ID":"46e82c61-c992-432e-b1ab-dffdaadccf70","Type":"ContainerStarted","Data":"bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59"} Oct 30 18:00:38.844483 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.844457 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" event={"ID":"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03","Type":"ContainerStarted","Data":"ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49"} Oct 30 18:00:38.848414 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.848389 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" event={"ID":"47a7c100-1107-4314-aa33-526e92b42c1f","Type":"ContainerStarted","Data":"167ac39baeb723a58def8d9ab2747ae398b523efd70cd9cb996dff2d58b76f05"} Oct 30 18:00:38.848527 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.848433 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" event={"ID":"47a7c100-1107-4314-aa33-526e92b42c1f","Type":"ContainerStarted","Data":"84757f5b4061c8bb5fe1621396ffe703073824f1e5c7b90515b97e377d157bed"} Oct 30 18:00:38.850471 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.850430 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerStarted","Data":"76cd0c09677ddeab12e30a606c5836ded0a8c0182b0371cc7b9c092257c95a88"} Oct 30 18:00:38.868163 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.868110 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" podStartSLOduration=430.868095304 podStartE2EDuration="7m10.868095304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:38.866775755 +0000 UTC m=+31.760002145" watchObservedRunningTime="2026-10-30 18:00:38.868095304 +0000 UTC m=+31.761321698" Oct 30 18:00:38.951538 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:38.951480 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" podStartSLOduration=430.951461088 podStartE2EDuration="7m10.951461088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:38.911231264 +0000 UTC m=+31.804457659" watchObservedRunningTime="2026-10-30 18:00:38.951461088 +0000 UTC m=+31.844687481" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.147393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.147632 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.147612773 +0000 UTC m=+34.040839184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : configmap references non-existent config key: service-ca.crt Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148061 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148115 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.148099785 +0000 UTC m=+34.041326162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148198 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148209 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f75c69b8c-gmvng: secret "image-registry-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148243 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls podName:3f409e7d-56d0-43ed-ac57-5724283f1a57 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.148231538 +0000 UTC m=+34.041457912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls") pod "image-registry-6f75c69b8c-gmvng" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57") : secret "image-registry-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148300 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148330 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls podName:2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.148320412 +0000 UTC m=+34.041546799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nmrh6" (UID: "2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd") : secret "samples-operator-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148385 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:39.148560 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148394 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bf557f7d-4tfww: secret "image-registry-tls" not found Oct 30 18:00:39.149480 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148424 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls podName:8528ac59-25e8-435f-9d93-599e19af51c5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.1484129 +0000 UTC m=+34.041639286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls") pod "image-registry-85bf557f7d-4tfww" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5") : secret "image-registry-tls" not found Oct 30 18:00:39.149480 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148475 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:00:39.149480 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.148503 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.148493949 +0000 UTC m=+34.041720334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : secret "router-metrics-certs-default" not found Oct 30 18:00:39.248399 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.248361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:39.248583 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:39.248469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:39.248693 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.248626 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:00:39.248693 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.248686 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls podName:3208b565-97ac-4ed9-9d36-a289544c5fd1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.248668576 +0000 UTC m=+34.141894962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls") pod "dns-default-fnrlr" (UID: "3208b565-97ac-4ed9-9d36-a289544c5fd1") : secret "dns-default-metrics-tls" not found Oct 30 18:00:39.248807 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.248763 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:39.248807 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:39.248799 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:41.248788419 +0000 UTC m=+34.142014804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:40.861618 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:40.861291 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" event={"ID":"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03","Type":"ContainerStarted","Data":"f145cd4899b99161ebfbbe2fae7dbfb570a224d9577d1820f8066023aee47db5"} Oct 30 18:00:40.900232 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:40.900173 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" podStartSLOduration=432.900153277 podStartE2EDuration="7m12.900153277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:38.952896368 +0000 UTC m=+31.846122754" watchObservedRunningTime="2026-10-30 18:00:40.900153277 +0000 UTC m=+33.793379663" Oct 30 18:00:40.901673 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:40.901632 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" podStartSLOduration=401.440997005 podStartE2EDuration="6m43.901622413s" totalImagesPullingTime="2.460625408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:40.899625882 +0000 UTC m=+33.792852307" watchObservedRunningTime="2026-10-30 18:00:40.901622413 +0000 UTC m=+33.794848809" Oct 30 18:00:41.006688 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.006545 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dw7mk_b440ae4e-411c-48fd-a8ed-45596e6b4147/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:41.097983 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.097948 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dw7mk_b440ae4e-411c-48fd-a8ed-45596e6b4147/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:41.170201 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:41.170382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:41.170382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:41.170382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:41.170382 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:41.170575 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.170395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:41.170575 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170561 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.1705417 +0000 UTC m=+38.063768086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : configmap references non-existent config key: service-ca.crt Oct 30 18:00:41.170710 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170681 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:00:41.170760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170715 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:41.170760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170734 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f75c69b8c-gmvng: secret "image-registry-tls" not found Oct 30 18:00:41.170760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170681 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:41.170760 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170752 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170737 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls podName:2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.170724034 +0000 UTC m=+38.063950418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nmrh6" (UID: "2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd") : secret "samples-operator-tls" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170800 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.170781985 +0000 UTC m=+38.064008360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170806 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170813 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bf557f7d-4tfww: secret "image-registry-tls" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170819 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls podName:3f409e7d-56d0-43ed-ac57-5724283f1a57 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.170807855 +0000 UTC m=+38.064034232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls") pod "image-registry-6f75c69b8c-gmvng" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57") : secret "image-registry-tls" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170835 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.170826222 +0000 UTC m=+38.064052599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : secret "router-metrics-certs-default" not found Oct 30 18:00:41.170926 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.170849 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls podName:8528ac59-25e8-435f-9d93-599e19af51c5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.170841381 +0000 UTC m=+38.064067771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls") pod "image-registry-85bf557f7d-4tfww" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5") : secret "image-registry-tls" not found Oct 30 18:00:41.271840 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.271744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:41.272013 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.271894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:41.272013 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.271908 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:41.272013 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.271985 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.271965877 +0000 UTC m=+38.165192265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:41.272178 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.272046 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:00:41.272178 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.272105 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls podName:3208b565-97ac-4ed9-9d36-a289544c5fd1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.27209015 +0000 UTC m=+38.165316535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls") pod "dns-default-fnrlr" (UID: "3208b565-97ac-4ed9-9d36-a289544c5fd1") : secret "dns-default-metrics-tls" not found Oct 30 18:00:41.373282 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.373240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:41.373451 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.373295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:41.373508 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.373463 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 18:00:41.373554 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:41.373525 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs podName:655bf5c4-4149-4e0e-b03b-10a41946b0f1 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:13.373507117 +0000 UTC m=+66.266733692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs") pod "network-metrics-daemon-79lsr" (UID: "655bf5c4-4149-4e0e-b03b-10a41946b0f1") : secret "metrics-daemon-secret" not found Oct 30 18:00:41.383041 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.383010 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:41.474647 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.474552 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:41.478406 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.478376 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:41.576912 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.576656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:41.618673 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.618642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:41.932451 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.932417 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jzwlg"] Oct 30 18:00:41.944381 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:41.944345 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca34023_3797_4998_9674_ee14daf13450.slice/crio-198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b WatchSource:0}: Error finding container 198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b: Status 404 returned error can't find the container with id 198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b Oct 30 18:00:41.985033 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:41.985002 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tn9m"] Oct 30 18:00:42.874949 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:42.874910 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tn9m" event={"ID":"a7517a84-07c0-4210-9303-ac839cdd188b","Type":"ContainerStarted","Data":"d13a2b8fad05bf837c07fce7cd1b4888727c8567efe0c6deb43ba94ebf70abe4"} Oct 30 18:00:42.876223 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:42.876192 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jzwlg" event={"ID":"dca34023-3797-4998-9674-ee14daf13450","Type":"ContainerStarted","Data":"198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b"} Oct 30 18:00:43.884277 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.883538 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" event={"ID":"891a4e9b-107c-45f6-bef6-d825fe05e079","Type":"ContainerStarted","Data":"e390538830a11d135759decd8d6a556a6d10d1d45e818317aa099d0232fb17ac"} Oct 30 18:00:43.887027 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.886997 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/0.log" Oct 30 18:00:43.887134 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.887040 2569 generic.go:356] "Generic (PLEG): container finished" podID="d861baa6-45e0-44b6-9336-1ed846e00b49" containerID="d2701afa08f67ac886c393248187c144794fd18836bc8a0bcc2d41596e0f3396" exitCode=255 Oct 30 18:00:43.888063 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.888043 2569 scope.go:122] "RemoveContainer" containerID="d2701afa08f67ac886c393248187c144794fd18836bc8a0bcc2d41596e0f3396" Oct 30 18:00:43.888413 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.888309 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" event={"ID":"d861baa6-45e0-44b6-9336-1ed846e00b49","Type":"ContainerDied","Data":"d2701afa08f67ac886c393248187c144794fd18836bc8a0bcc2d41596e0f3396"} Oct 30 18:00:43.890291 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.890248 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" event={"ID":"46e82c61-c992-432e-b1ab-dffdaadccf70","Type":"ContainerStarted","Data":"c7c2ee3e1af0166c85eb9927302a28cbe393a824f4f3e7ed8ddde8f2208e7781"} Oct 30 18:00:43.892840 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.892815 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerStarted","Data":"319e4e2093f1b430e7a74e76057c28e0876750221f102924a7a1b333c571dd84"} Oct 30 18:00:43.927763 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.926845 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" podStartSLOduration=401.874867095 podStartE2EDuration="6m46.926826738s" totalImagesPullingTime="5.051959643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:43.92626493 +0000 UTC m=+36.819491326" watchObservedRunningTime="2026-10-30 18:00:43.926826738 +0000 UTC m=+36.820053136" Oct 30 18:00:43.967798 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:43.967199 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" podStartSLOduration=401.910395363 podStartE2EDuration="6m46.967177398s" totalImagesPullingTime="5.056782035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:43.966588862 +0000 UTC m=+36.859815256" watchObservedRunningTime="2026-10-30 18:00:43.967177398 +0000 UTC m=+36.860403795" Oct 30 18:00:44.010903 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.009302 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" podStartSLOduration=401.974895871 podStartE2EDuration="6m47.00928426s" totalImagesPullingTime="5.034388389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:44.008543024 +0000 UTC m=+36.901769420" watchObservedRunningTime="2026-10-30 18:00:44.00928426 +0000 UTC m=+36.902510654" Oct 30 18:00:44.150873 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.150838 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9"] Oct 30 18:00:44.176264 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.176223 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn"] Oct 30 18:00:44.177742 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.176543 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:44.196218 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.195870 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-njkgb\"" Oct 30 18:00:44.196218 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.195969 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:44.196218 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.196037 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:00:44.198966 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.198768 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9"] Oct 30 18:00:44.198966 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.198813 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn"] Oct 30 18:00:44.198966 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.198906 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:44.218325 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.218299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5j4v\" (UniqueName: \"kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v\") pod \"migrator-66df5d496b-mzqd9\" (UID: \"e0bcf238-c8de-42ac-a905-348017d59ff8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:44.218532 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.218426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmqk\" (UniqueName: \"kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk\") pod \"migrator-66df5d496b-v8jsn\" (UID: \"cb64abf9-519f-4791-b5fc-645b72981f53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:44.246321 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.246289 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7bvmg"] Oct 30 18:00:44.280502 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.279459 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bvmg"] Oct 30 18:00:44.280502 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.279626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.289087 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.289064 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:00:44.295688 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.295646 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:00:44.296034 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.295840 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:00:44.296219 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.296085 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:00:44.305313 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.305289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.307886 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.307837 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:00:44.320811 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.320781 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmqk\" (UniqueName: \"kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk\") pod \"migrator-66df5d496b-v8jsn\" (UID: \"cb64abf9-519f-4791-b5fc-645b72981f53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:44.320906 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.320834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.320906 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.320862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.320982 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.320906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.321039 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.320955 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.321079 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j4v\" (UniqueName: \"kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v\") pod \"migrator-66df5d496b-mzqd9\" (UID: \"e0bcf238-c8de-42ac-a905-348017d59ff8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:44.321143 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.321219 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.322016 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.322016 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmq9\" (UniqueName: \"kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.322016 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.321394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.342132 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.341831 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmqk\" (UniqueName: \"kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk\") pod \"migrator-66df5d496b-v8jsn\" (UID: \"cb64abf9-519f-4791-b5fc-645b72981f53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:44.344675 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.344623 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j4v\" (UniqueName: \"kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v\") pod \"migrator-66df5d496b-mzqd9\" (UID: \"e0bcf238-c8de-42ac-a905-348017d59ff8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:44.422628 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422577 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.422833 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.422833 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmq9\" (UniqueName: \"kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.422833 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.422833 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.422833 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.423041 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422863 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.423041 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.423041 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.422991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.423401 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.423379 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.423526 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.423441 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:00:44.423526 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.423495 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:44.923475895 +0000 UTC m=+37.816702271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : configmap "kube-rbac-proxy" not found Oct 30 18:00:44.424193 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.424173 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:44.424258 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.424236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.424311 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.424278 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:44.92425752 +0000 UTC m=+37.817483910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:44.424435 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.424413 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.424561 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.424544 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.424906 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.424885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.453627 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.450073 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmq9\" (UniqueName: \"kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.453627 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.450230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.513348 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.513310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:44.536136 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.536101 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:44.682121 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.682088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:44.860314 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.860236 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9"] Oct 30 18:00:44.866547 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:44.866518 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bcf238_c8de_42ac_a905_348017d59ff8.slice/crio-c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2 WatchSource:0}: Error finding container c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2: Status 404 returned error can't find the container with id c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2 Oct 30 18:00:44.895046 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.895019 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn"] Oct 30 18:00:44.898443 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:44.898405 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb64abf9_519f_4791_b5fc_645b72981f53.slice/crio-f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4 WatchSource:0}: Error finding container f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4: Status 404 returned error can't find the container with id f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4 Oct 30 18:00:44.899269 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.899232 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dq8ks" event={"ID":"47517669-cee7-4488-827e-b605f1cbfbc0","Type":"ContainerStarted","Data":"d752e1d54a90a43ce5beb1c7fd75d78c810e6b0f3d44f9ea2d5355101b3be349"} Oct 30 18:00:44.901864 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.901786 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:44.902346 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.902325 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/0.log" Oct 30 18:00:44.902438 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.902370 2569 generic.go:356] "Generic (PLEG): container finished" podID="d861baa6-45e0-44b6-9336-1ed846e00b49" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" exitCode=255 Oct 30 18:00:44.902540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.902483 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" event={"ID":"d861baa6-45e0-44b6-9336-1ed846e00b49","Type":"ContainerDied","Data":"a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f"} Oct 30 18:00:44.902540 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.902528 2569 scope.go:122] "RemoveContainer" containerID="d2701afa08f67ac886c393248187c144794fd18836bc8a0bcc2d41596e0f3396" Oct 30 18:00:44.902735 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.902697 2569 scope.go:122] "RemoveContainer" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" Oct 30 18:00:44.902897 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.902877 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ts48c_openshift-console-operator(d861baa6-45e0-44b6-9336-1ed846e00b49)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" podUID="d861baa6-45e0-44b6-9336-1ed846e00b49" Oct 30 18:00:44.905484 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.905463 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" event={"ID":"0239d118-e691-4ac3-9e13-e1ced388a733","Type":"ContainerStarted","Data":"58f5c60b721fcca1b75146454c8ea9781ae61fc538c7f07757f321d1374e834f"} Oct 30 18:00:44.907048 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.907004 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" event={"ID":"e0bcf238-c8de-42ac-a905-348017d59ff8","Type":"ContainerStarted","Data":"c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2"} Oct 30 18:00:44.928375 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.928338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.928526 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.928461 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:44.928684 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.928664 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:44.928963 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.928939 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.928919672 +0000 UTC m=+38.822146049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:44.929019 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.928964 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:00:44.929252 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:44.929237 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:45.929219205 +0000 UTC m=+38.822445598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : configmap "kube-rbac-proxy" not found Oct 30 18:00:44.939088 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.939042 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dq8ks" podStartSLOduration=2.434538567 podStartE2EDuration="7.939030191s" totalImagesPullingTime="5.504491624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:44.938782216 +0000 UTC m=+37.832008606" watchObservedRunningTime="2026-10-30 18:00:44.939030191 +0000 UTC m=+37.832256618" Oct 30 18:00:44.975884 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.975852 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6qxt6"] Oct 30 18:00:44.982972 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:44.982924 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" podStartSLOduration=402.409111496 podStartE2EDuration="6m47.982911251s" totalImagesPullingTime="5.573799755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:44.980993165 +0000 UTC m=+37.874219557" watchObservedRunningTime="2026-10-30 18:00:44.982911251 +0000 UTC m=+37.876137644" Oct 30 18:00:45.015811 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.015778 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6qxt6"] Oct 30 18:00:45.015940 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.015888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.026488 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.026256 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:00:45.029256 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.029235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.029443 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.029426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9g69\" (UniqueName: \"kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.029492 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.029454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.032805 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.032786 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:00:45.032896 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.032885 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:00:45.036854 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.034684 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j6bp8\"" Oct 30 18:00:45.036854 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.034753 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:00:45.036854 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.034828 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:00:45.131151 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.131053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.131303 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.131238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9g69\" (UniqueName: \"kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.131303 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.131261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.131908 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.131860 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.134561 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.134537 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.151962 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.151935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9g69\" (UniqueName: \"kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.232413 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:45.232574 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232482 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:45.232574 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:45.232574 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232563 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.232538636 +0000 UTC m=+46.125765027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : configmap references non-existent config key: service-ca.crt Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232642 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232643 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232662 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f75c69b8c-gmvng: secret "image-registry-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232688 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.232675499 +0000 UTC m=+46.125901886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232709 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls podName:3f409e7d-56d0-43ed-ac57-5724283f1a57 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.232700195 +0000 UTC m=+46.125926572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls") pod "image-registry-6f75c69b8c-gmvng" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57") : secret "image-registry-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232740 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232751 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85bf557f7d-4tfww: secret "image-registry-tls" not found Oct 30 18:00:45.232799 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232784 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls podName:8528ac59-25e8-435f-9d93-599e19af51c5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.232773388 +0000 UTC m=+46.125999778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls") pod "image-registry-85bf557f7d-4tfww" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5") : secret "image-registry-tls" not found Oct 30 18:00:45.233295 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.232808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:45.233295 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232838 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:00:45.233295 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232872 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls podName:2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.23286222 +0000 UTC m=+46.126088621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nmrh6" (UID: "2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd") : secret "samples-operator-tls" not found Oct 30 18:00:45.233295 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232894 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:00:45.233295 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.232924 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs podName:b78ec231-1951-451f-8ec8-b70e20b98fc5 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.232914722 +0000 UTC m=+46.126141133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs") pod "router-default-7765667bbf-4zpgd" (UID: "b78ec231-1951-451f-8ec8-b70e20b98fc5") : secret "router-metrics-certs-default" not found Oct 30 18:00:45.334266 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.334231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:45.334445 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.334370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:45.334445 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.334427 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:00:45.334539 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.334506 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls podName:3208b565-97ac-4ed9-9d36-a289544c5fd1 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.334485274 +0000 UTC m=+46.227711668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls") pod "dns-default-fnrlr" (UID: "3208b565-97ac-4ed9-9d36-a289544c5fd1") : secret "dns-default-metrics-tls" not found Oct 30 18:00:45.334539 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.334505 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:45.334642 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.334556 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:53.334544406 +0000 UTC m=+46.227770779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:45.344364 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.344340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:45.666566 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.666434 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6qxt6"] Oct 30 18:00:45.669473 ip-10-0-143-94 kubenswrapper[2569]: W1030 18:00:45.669439 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d95a5a_9823_4002_a14c_c1c42f2cdbe4.slice/crio-84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e WatchSource:0}: Error finding container 84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e: Status 404 returned error can't find the container with id 84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e Oct 30 18:00:45.914754 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.914659 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" event={"ID":"75d95a5a-9823-4002-a14c-c1c42f2cdbe4","Type":"ContainerStarted","Data":"7eff5db009928bad0940b9a64b51cc7276acd125b67fd7533b3b769096d436ce"} Oct 30 18:00:45.914754 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.914718 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" event={"ID":"75d95a5a-9823-4002-a14c-c1c42f2cdbe4","Type":"ContainerStarted","Data":"84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e"} Oct 30 18:00:45.918411 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.918313 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" event={"ID":"db09f543-253e-47ae-81bf-977c1e6323b7","Type":"ContainerStarted","Data":"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28"} Oct 30 18:00:45.918411 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.918367 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" event={"ID":"db09f543-253e-47ae-81bf-977c1e6323b7","Type":"ContainerStarted","Data":"908d48a8d7412e8ad1ce496132da0ec5a51c7102e57de3c7cfe9848a33d978fc"} Oct 30 18:00:45.919836 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.919807 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" event={"ID":"cb64abf9-519f-4791-b5fc-645b72981f53","Type":"ContainerStarted","Data":"f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4"} Oct 30 18:00:45.921999 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.921969 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tn9m" event={"ID":"a7517a84-07c0-4210-9303-ac839cdd188b","Type":"ContainerStarted","Data":"8e2b5a5f9fd258bca1257fd723e6592f09acaaa50c8ec3e8d6ad8732b6ca7e5c"} Oct 30 18:00:45.922493 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.922474 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:45.926352 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.926335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:45.927060 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.927044 2569 scope.go:122] "RemoveContainer" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" Oct 30 18:00:45.927205 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.927189 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ts48c_openshift-console-operator(d861baa6-45e0-44b6-9336-1ed846e00b49)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" podUID="d861baa6-45e0-44b6-9336-1ed846e00b49" Oct 30 18:00:45.940655 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.940390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:45.940655 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.940566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:45.940939 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.940900 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:00:45.940999 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.940974 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:47.940957027 +0000 UTC m=+40.834183403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : configmap "kube-rbac-proxy" not found Oct 30 18:00:45.941746 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.941378 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:45.941746 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:45.941433 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:47.941415816 +0000 UTC m=+40.834642201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:45.961610 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:45.961535 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" podStartSLOduration=1.961499503 podStartE2EDuration="1.961499503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:45.959307433 +0000 UTC m=+38.852533827" watchObservedRunningTime="2026-10-30 18:00:45.961499503 +0000 UTC m=+38.854725902" Oct 30 18:00:46.097329 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:46.097260 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2tn9m" podStartSLOduration=34.290642195 podStartE2EDuration="38.097240077s" totalImagesPullingTime="3.806597882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:46.056560117 +0000 UTC m=+38.949786510" watchObservedRunningTime="2026-10-30 18:00:46.097240077 +0000 UTC m=+38.990466471" Oct 30 18:00:46.098217 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:46.098174 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" podStartSLOduration=2.098163864 podStartE2EDuration="2.098163864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:46.096821676 +0000 UTC m=+38.990048082" watchObservedRunningTime="2026-10-30 18:00:46.098163864 +0000 UTC m=+38.991390257" Oct 30 18:00:46.932780 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:46.932712 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" event={"ID":"e0bcf238-c8de-42ac-a905-348017d59ff8","Type":"ContainerStarted","Data":"f1a0aad1b0671ae8987a73a80094adb926e93b2c4c9dbb4d926717fefee2c5d9"} Oct 30 18:00:46.934851 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:46.934431 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" event={"ID":"cb64abf9-519f-4791-b5fc-645b72981f53","Type":"ContainerStarted","Data":"5aa63412546e64edff7b2d47bd8423865aaf72fda02a05cf54e35b6b898e5d5b"} Oct 30 18:00:46.972619 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:46.972537 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" podStartSLOduration=1.244833248 podStartE2EDuration="2.972520629s" totalImagesPullingTime="1.727687381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:46.972440188 +0000 UTC m=+39.865666583" watchObservedRunningTime="2026-10-30 18:00:46.972520629 +0000 UTC m=+39.865747023" Oct 30 18:00:47.016406 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.016347 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" podStartSLOduration=1.2928379890000001 podStartE2EDuration="3.016328235s" totalImagesPullingTime="1.723490246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:47.014866098 +0000 UTC m=+39.908092495" watchObservedRunningTime="2026-10-30 18:00:47.016328235 +0000 UTC m=+39.909554632" Oct 30 18:00:47.849631 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.849571 2569 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:47.849835 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.849721 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:47.850126 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.850111 2569 scope.go:122] "RemoveContainer" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" Oct 30 18:00:47.850322 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:47.850299 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ts48c_openshift-console-operator(d861baa6-45e0-44b6-9336-1ed846e00b49)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" podUID="d861baa6-45e0-44b6-9336-1ed846e00b49" Oct 30 18:00:47.962557 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.962517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:47.963054 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:47.962677 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:00:47.963054 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:47.962729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:47.963054 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:47.962750 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.962730352 +0000 UTC m=+44.855956739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : configmap "kube-rbac-proxy" not found Oct 30 18:00:47.963054 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:47.962816 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:47.963054 ip-10-0-143-94 kubenswrapper[2569]: E1030 18:00:47.962863 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.962848038 +0000 UTC m=+44.856074414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:48.943187 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:48.943141 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jzwlg" event={"ID":"dca34023-3797-4998-9674-ee14daf13450","Type":"ContainerStarted","Data":"24025b3ec03c936d1a7d74285ca92f971352b6409b3c458a6e897ec8710c93e1"} Oct 30 18:00:48.981935 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:48.981885 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jzwlg" podStartSLOduration=34.133721315 podStartE2EDuration="40.981870821s" totalImagesPullingTime="6.848149506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:48.980967522 +0000 UTC m=+41.874193915" watchObservedRunningTime="2026-10-30 18:00:48.981870821 +0000 UTC m=+41.875097210" Oct 30 18:00:49.103407 ip-10-0-143-94 kubenswrapper[2569]: I1030 18:00:49.103287 2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:49.103440 ip-10-0-143-94 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:00:49.113989 ip-10-0-143-94 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:00:49.114165 ip-10-0-143-94 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:00:49.114221 ip-10-0-143-94 systemd[1]: kubelet.service: Consumed 3.297s CPU time. Oct 30 18:00:49.125157 ip-10-0-143-94 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:00:49.218273 ip-10-0-143-94 kubenswrapper[5024]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:49.218273 ip-10-0-143-94 kubenswrapper[5024]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:00:49.218273 ip-10-0-143-94 kubenswrapper[5024]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:49.218273 ip-10-0-143-94 kubenswrapper[5024]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:49.220364 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220282 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.220364 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220364 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220369 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220373 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220375 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220379 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220382 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220388 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220391 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220394 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220397 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220400 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220402 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220405 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220408 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220410 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220413 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220416 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220418 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220421 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220424 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.220440 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220427 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220430 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220432 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220435 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220438 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220441 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220443 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220446 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220449 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220452 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220455 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220458 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220461 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220463 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220466 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220469 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220471 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220474 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220478 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.220972 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220481 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220485 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220490 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220493 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220497 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220500 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220502 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220505 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220507 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220510 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220513 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220515 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220518 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220520 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220523 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220528 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220531 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220533 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220536 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.221461 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220539 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220541 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220544 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220547 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220549 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220552 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220554 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220557 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220559 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220562 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220565 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220567 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220570 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220573 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220575 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220578 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220581 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220583 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220586 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.221957 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220588 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220613 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220617 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220620 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220623 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220626 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220628 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220631 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220634 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220637 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220640 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220642 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220645 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220648 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220650 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220653 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220656 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220658 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220661 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220664 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.222456 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220666 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220670 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220672 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220676 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220678 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220681 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220684 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.220687 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221432 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221439 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221442 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221445 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221448 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221451 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221454 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221457 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221459 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221462 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221465 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221469 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.222977 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221472 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221475 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221477 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221480 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221483 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221485 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221488 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221490 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221493 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221496 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221498 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221501 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221503 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221506 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221509 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221511 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221514 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221516 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221519 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.223517 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221522 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221525 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221528 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221530 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221533 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221535 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221538 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221540 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221543 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221545 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221548 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221551 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221554 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221556 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221559 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221561 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221565 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221568 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221571 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221574 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.224027 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221577 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221579 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221583 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221587 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221590 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221608 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221611 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221613 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221616 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221618 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221621 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221625 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221627 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221630 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221633 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221636 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221639 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221641 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221644 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221646 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.224539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221649 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221652 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221655 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221657 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221660 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221663 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221665 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221668 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221671 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221674 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221677 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221679 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221682 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221684 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221687 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221690 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221692 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221695 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221697 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.225099 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221700 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221702 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221705 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221709 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221712 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221715 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221718 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221721 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221723 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221732 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221735 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221737 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221740 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221742 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221745 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.221749 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222243 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222248 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222251 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.225577 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222254 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222256 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222259 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222261 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222264 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222266 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222269 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222272 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222275 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222277 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222280 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222283 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222285 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222288 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222290 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222293 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222295 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222298 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222301 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.226067 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222304 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222307 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222310 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222316 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222318 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222321 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222323 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222326 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222328 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222331 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222333 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222336 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222338 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222341 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222343 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222346 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222349 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222351 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222354 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222357 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222359 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.226738 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222362 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222364 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222367 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222370 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222372 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222375 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222378 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222381 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222383 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222386 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222389 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222391 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222394 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222396 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222401 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222403 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222406 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222408 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222411 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.227685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222413 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222416 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222419 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222421 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222424 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222426 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222429 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222431 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222434 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222436 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222439 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222441 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222444 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222446 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222449 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222452 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222454 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222457 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222460 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222463 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.228527 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222465 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222468 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222471 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222474 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222476 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222479 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222481 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222485 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222488 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222491 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222493 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222496 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222498 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222501 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222505 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222509 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222512 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222515 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222518 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.229429 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222520 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222524 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222528 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222530 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.222533 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222642 5024 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222649 5024 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222655 5024 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222660 5024 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222665 5024 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222668 5024 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222672 5024 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222677 5024 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222680 5024 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222684 5024 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222687 5024 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222690 5024 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222693 5024 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222696 5024 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222699 5024 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222705 5024 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222707 5024 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222713 5024 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:00:49.230346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222717 5024 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222720 5024 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222723 5024 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222726 5024 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222730 5024 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222734 5024 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222737 5024 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222740 5024 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222745 5024 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222750 5024 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222754 5024 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222759 5024 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222763 5024 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222768 5024 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222774 5024 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222779 5024 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222783 5024 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222786 5024 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222789 5024 flags.go:64] FLAG: --enable-server="true" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222792 5024 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222795 5024 flags.go:64] FLAG: --event-burst="100" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222799 5024 flags.go:64] FLAG: --event-qps="50" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222802 5024 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222805 5024 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:00:49.231388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222808 5024 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222812 5024 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222815 5024 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222818 5024 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222821 5024 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222827 5024 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222832 5024 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222840 5024 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222845 5024 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222850 5024 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222853 5024 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222856 5024 flags.go:64] FLAG: --feature-gates="" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222860 5024 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222864 5024 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222867 5024 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222870 5024 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222873 5024 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222876 5024 flags.go:64] FLAG: --help="false" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222879 5024 flags.go:64] FLAG: --hostname-override="ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222882 5024 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222886 5024 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222889 5024 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222892 5024 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:00:49.232403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222895 5024 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222898 5024 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222901 5024 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222904 5024 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222907 5024 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222911 5024 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222916 5024 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222921 5024 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222926 5024 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222930 5024 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222935 5024 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222940 5024 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222943 5024 flags.go:64] FLAG: --lock-file="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222946 5024 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222950 5024 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222953 5024 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222961 5024 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222964 5024 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222968 5024 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222972 5024 flags.go:64] FLAG: --logging-format="text" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222975 5024 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222978 5024 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222981 5024 flags.go:64] FLAG: --manifest-url="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222984 5024 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222989 5024 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:00:49.233145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222992 5024 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222996 5024 flags.go:64] FLAG: --max-pods="110" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.222999 5024 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223003 5024 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223008 5024 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223013 5024 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223018 5024 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223022 5024 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223025 5024 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223032 5024 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223035 5024 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223038 5024 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223042 5024 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223044 5024 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223047 5024 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223050 5024 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223053 5024 flags.go:64] FLAG: --port="10250" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223057 5024 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223060 5024 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0681bd3f001fe1f10" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223063 5024 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223066 5024 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223070 5024 flags.go:64] FLAG: --register-node="true" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223073 5024 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223079 5024 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223082 5024 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223085 5024 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:00:49.233783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223091 5024 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223097 5024 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223102 5024 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223106 5024 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223109 5024 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223112 5024 flags.go:64] FLAG: --runonce="false" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223115 5024 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223118 5024 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223120 5024 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223123 5024 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223126 5024 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223129 5024 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223132 5024 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223135 5024 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223138 5024 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223141 5024 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223144 5024 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223146 5024 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223150 5024 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223152 5024 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223155 5024 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223161 5024 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223164 5024 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223169 5024 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223175 5024 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:00:49.234434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223180 5024 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223185 5024 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223189 5024 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223192 5024 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223197 5024 flags.go:64] FLAG: --v="2" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223201 5024 flags.go:64] FLAG: --version="false" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223205 5024 flags.go:64] FLAG: --vmodule="" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223209 5024 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223213 5024 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223364 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223371 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223374 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223377 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223380 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223383 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223386 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223388 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223391 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223393 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223396 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223399 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.235068 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223401 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223404 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223407 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223411 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223416 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223421 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223425 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223429 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223431 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223434 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223436 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223439 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223442 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223447 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223449 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223453 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223456 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223458 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223461 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223464 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.235772 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223467 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223470 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223472 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223475 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223477 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223480 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223483 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223485 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223489 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223494 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223498 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223503 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223507 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223509 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223512 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223514 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223517 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223520 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223523 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223526 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.236351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223529 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223531 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223534 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223536 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223539 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223543 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223546 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223549 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223553 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223557 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223560 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223563 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223565 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223569 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223574 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223578 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223582 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223587 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223590 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223607 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.236908 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223610 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223613 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223616 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223618 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223621 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223623 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223626 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223629 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223631 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223634 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223636 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223639 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223642 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223644 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223647 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223649 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223654 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223660 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223665 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.237411 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223670 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223673 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223676 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223678 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223681 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223684 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223686 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223690 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223693 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223695 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223698 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223700 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223703 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223706 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223708 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.223738 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:49.237922 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223776 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223782 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223786 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223790 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223793 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223795 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223798 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223801 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223804 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223806 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223810 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223814 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223819 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223824 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223830 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223835 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223838 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223842 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223845 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.238332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223848 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223850 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223853 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223855 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223858 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223861 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223864 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223867 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223869 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223872 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223874 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223877 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223879 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223882 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223884 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223887 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223889 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223892 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223897 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223901 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.238944 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223906 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223910 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223914 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223918 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223920 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223923 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223925 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223929 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223932 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223934 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223938 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223941 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223943 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223946 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223949 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223951 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223954 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223957 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223960 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223962 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.239510 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223965 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223969 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223973 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223976 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223978 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223981 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223984 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223986 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223989 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223991 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223994 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223997 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.223999 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224002 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224004 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224007 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224009 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224012 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224014 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224018 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.240066 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224022 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224024 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224029 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224032 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224034 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224037 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224040 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224042 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224045 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224048 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224050 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224053 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224056 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224059 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224061 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224064 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224066 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224069 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224071 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.240902 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224074 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224076 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224079 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224081 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224083 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224086 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224088 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.224091 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.229054 5024 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.229074 5024 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.229098 5024 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.229105 5024 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229184 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229192 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229197 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229201 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229206 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229211 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229215 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229220 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.241387 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229224 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229229 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229233 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229237 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229242 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229246 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229250 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229254 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229259 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229264 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229269 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229273 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229278 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229282 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229287 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229291 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229295 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229299 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229307 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.241913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229313 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229318 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229324 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229328 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229332 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229339 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229344 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229349 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229355 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229359 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229364 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229369 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229374 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229378 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229382 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229386 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229391 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229395 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229399 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.242379 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229404 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229409 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229413 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229417 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229421 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229426 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229430 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229434 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229438 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229442 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229446 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229450 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229454 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229459 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229463 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229468 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229473 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229477 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229482 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229486 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.242872 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229491 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229495 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229500 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229504 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229508 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229513 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229517 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229522 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229526 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229530 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229535 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229539 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229543 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229547 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229551 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229556 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229560 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229564 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229568 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229573 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.243351 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229577 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229581 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229586 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229590 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229615 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229621 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229625 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229629 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229633 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229638 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229643 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229647 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229651 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229655 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229659 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229663 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229667 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229671 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229676 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.243864 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229680 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.229722 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229929 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229936 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229941 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229945 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229950 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229954 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229958 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229962 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229966 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229971 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229975 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229979 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229983 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229988 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.244473 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229992 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.229996 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230000 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230004 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230009 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230013 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230017 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230022 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230027 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230031 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230035 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230040 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230044 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230048 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230052 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230058 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230064 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230068 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230073 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.245059 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230077 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230081 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230085 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230089 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230093 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230098 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230102 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230106 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230110 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230114 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230119 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230123 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230127 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230131 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230135 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230139 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230144 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230148 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230151 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.245630 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230158 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230164 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230170 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230175 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230179 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230183 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230188 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230192 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230196 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230200 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230204 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230208 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230212 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230216 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230219 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230223 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230228 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230232 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230236 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.246128 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230241 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230245 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230249 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230253 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230257 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230262 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230266 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230270 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230274 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230278 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230282 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230286 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230290 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230294 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230298 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230302 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230307 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230312 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230316 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230320 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.246644 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230324 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230328 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230332 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230337 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230341 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230345 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230348 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230353 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230357 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230361 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230365 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230369 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230373 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230377 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230381 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.230419 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:49.247144 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230496 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230504 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230510 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230515 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230520 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230524 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230528 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230532 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230536 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230541 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230545 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230549 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230553 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230559 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230563 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230568 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230572 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230576 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230580 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230585 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:49.247548 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230589 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230610 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230617 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230624 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230633 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230639 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230645 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230649 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230654 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230658 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230663 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230667 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230671 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230675 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230680 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230684 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230688 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230693 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230697 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:49.248095 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230702 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230706 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230710 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230715 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230719 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230723 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230728 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230733 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230738 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230742 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230746 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230750 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230758 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230762 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230766 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230771 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230775 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230781 5024 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230785 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230789 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:49.248591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230793 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230797 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230802 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230806 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230811 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230815 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230819 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230825 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230829 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230833 5024 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230837 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230842 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230847 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230851 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230856 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230860 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230864 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230868 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230873 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230877 5024 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:49.249105 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230882 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230887 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230891 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230895 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230901 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230906 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230910 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230914 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230918 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230927 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230932 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230936 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230940 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230944 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230949 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230953 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230957 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230962 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230966 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:49.249581 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230970 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230974 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230978 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230983 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230987 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230992 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.230996 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:49.231003 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.231099 5024 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233422 5024 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233476 5024 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233710 5024 server.go:1014] "Starting client certificate rotation" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233851 5024 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233910 5024 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:07 +0000 UTC" deadline="2028-04-30 08:30:49.063896048 +0000 UTC" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.233942 5024 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13142h29m59.829955781s" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.234170 5024 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:49.250251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.235384 5024 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:49.250911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.236925 5024 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:00:49.250911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.238554 5024 log.go:25] "Validated CRI v1 image API" Oct 30 18:00:49.250911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.239316 5024 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:00:49.250911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.241347 5024 fs.go:135] Filesystem UUIDs: map[27391037-0853-4ffd-bd6b-d545fd5794e9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae54cb8d-68dd-45e9-85bd-60a90b363e59:/dev/nvme0n1p4] Oct 30 18:00:49.251103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.241362 5024 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cd9755409bd32ba6949dcd62cdd53155f4ed9dfec84003b02a8d10e22ebae58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cd9755409bd32ba6949dcd62cdd53155f4ed9dfec84003b02a8d10e22ebae58/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52511988ee22a11f593e0e82b46eaaa17fbf57b62ce424ece5e4b274a34adc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52511988ee22a11f593e0e82b46eaaa17fbf57b62ce424ece5e4b274a34adc09/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76cd0c09677ddeab12e30a606c5836ded0a8c0182b0371cc7b9c092257c95a88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76cd0c09677ddeab12e30a606c5836ded0a8c0182b0371cc7b9c092257c95a88/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84757f5b4061c8bb5fe1621396ffe703073824f1e5c7b90515b97e377d157bed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84757f5b4061c8bb5fe1621396ffe703073824f1e5c7b90515b97e377d157bed/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/908d48a8d7412e8ad1ce496132da0ec5a51c7102e57de3c7cfe9848a33d978fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/908d48a8d7412e8ad1ce496132da0ec5a51c7102e57de3c7cfe9848a33d978fc/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d13a2b8fad05bf837c07fce7cd1b4888727c8567efe0c6deb43ba94ebf70abe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d13a2b8fad05bf837c07fce7cd1b4888727c8567efe0c6deb43ba94ebf70abe4/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0239d118-e691-4ac3-9e13-e1ced388a733/volumes/kubernetes.io~projected/kube-api-access-mdb7l:{mountpoint:/var/lib/kubelet/pods/0239d118-e691-4ac3-9e13-e1ced388a733/volumes/kubernetes.io~projected/kube-api-access-mdb7l major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~projected/kube-api-access-t4sqc:{mountpoint:/var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~projected/kube-api-access-t4sqc major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2166c14d-2366-4925-9576-1fccab9df12c/volumes/kubernetes.io~projected/kube-api-access-fzx2r:{mountpoint:/var/lib/kubelet/pods/2166c14d-2366-4925-9576-1fccab9df12c/volumes/kubernetes.io~projected/kube-api-access-fzx2r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd/volumes/kubernetes.io~projected/kube-api-access-c4njc:{mountpoint:/var/lib/kubelet/pods/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd/volumes/kubernetes.io~projected/kube-api-access-c4njc major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3208b565-97ac-4ed9-9d36-a289544c5fd1/volumes/kubernetes.io~projected/kube-api-access-bwq2b:{mountpoint:/var/lib/kubelet/pods/3208b565-97ac-4ed9-9d36-a289544c5fd1/volumes/kubernetes.io~projected/kube-api-access-bwq2b major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b070856-f46c-4ebb-965c-d8aaacb18baf/volumes/kubernetes.io~projected/kube-api-access-v7hhd:{mountpoint:/var/lib/kubelet/pods/3b070856-f46c-4ebb-965c-d8aaacb18baf/volumes/kubernetes.io~projected/kube-api-access-v7hhd major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/kube-api-access-55cjl:{mountpoint:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/kube-api-access-55cjl major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~projected/kube-api-access-bht72:{mountpoint:/var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~projected/kube-api-access-bht72 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~projected/kube-api-access-nnxdr:{mountpoint:/var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~projected/kube-api-access-nnxdr major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47a7c100-1107-4314-aa33-526e92b42c1f/volumes/kubernetes.io~projected/kube-api-access-rqf6r:{mountpoint:/var/lib/kubelet/pods/47a7c100-1107-4314-aa33-526e92b42c1f/volumes/kubernetes.io~projected/kube-api-access-rqf6r major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5084064f-9031-4a3c-b74f-46abf27982a3/volumes/kubernetes.io~projected/kube-api-access-wx5cm:{mountpoint:/var/lib/kubelet/pods/5084064f-9031-4a3c-b74f-46abf27982a3/volumes/kubernetes.io~projected/kube-api-access-wx5cm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03/volumes/kubernetes.io~projected/kube-api-access-82fn6:{mountpoint:/var/lib/kubelet/pods/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03/volumes/kubernetes.io~projected/kube-api-access-82fn6 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/655bf5c4-4149-4e0e-b03b-10a41946b0f1/volumes/kubernetes.io~projected/kube-api-access-nsqxd:{mountpoint:/var/lib/kubelet/pods/655bf5c4-4149-4e0e-b03b-10a41946b0f1/volumes/kubernetes.io~projected/kube-api-access-nsqxd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~projected/kube-api-access-29qnt:{mountpoint:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~projected/kube-api-access-29qnt major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~projected/kube-api-access-k9g69:{mountpoint:/var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~projected/kube-api-access-k9g69 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~secret/signing-key major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d5af711-aeb9-4984-b639-1d6b13db2909/volumes/kubernetes.io~projected/kube-api-access-9g8xv:{mountpoint:/var/lib/kubelet/pods/7d5af711-aeb9-4984-b639-1d6b13db2909/volumes/kubernetes.io~projected/kube-api-access-9g8xv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dea29c6-0a9e-4f81-8237-f9199715657f/volumes/kubernetes.io~projected/kube-api-access-j7xbb:{mountpoint:/var/lib/kubelet/pods/7dea29c6-0a9e-4f81-8237-f9199715657f/volumes/kubernetes.io~projected/kube-api-access-j7xbb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/kube-api-access-npfv7:{mountpoint:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/kube-api-access-npfv7 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88b1723f-92ef-412a-a3ba-8742fa8b1131/volumes/kubernetes.io~projected/kube-api-access-wmmq9:{mountpoint:/var/lib/kubelet/pods/88b1723f-92ef-412a-a3ba-8742fa8b1131/volumes/kubernetes.io~projected/kube-api-access-wmmq9 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~projected/kube-api-access-dbkwc:{mountpoint:/var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~projected/kube-api-access-dbkwc major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf/volumes/kubernetes.io~projected/kube-api-access-b6mw8:{mountpoint:/var/lib/kubelet/pods/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf/volumes/kubernetes.io~projected/kube-api-access-b6mw8 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fe43c59-0934-4c18-84f1-381abdcf553d/volumes/kubernetes.io~projected/kube-api-access-5k227:{mountpoint:/var/lib/kubelet/pods/9fe43c59-0934-4c18-84f1-381abdcf553d/volumes/kubernetes.io~projected/kube-api-access-5k227 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7517a84-07c0-4210-9303-ac839cdd188b/volumes/kubernetes.io~projected/kube-api-access-mw28h:{mountpoint:/var/lib/kubelet/pods/a7517a84-07c0-4210-9303-ac839cdd188b/volumes/kubernetes.io~projected/kube-api-access-mw28h major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b440ae4e-411c-48fd-a8ed-45596e6b4147/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b440ae4e-411c-48fd-a8ed-45596e6b4147/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~projected/kube-api-access-xbksp:{mountpoint:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~projected/kube-api-access-xbksp major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb64abf9-519f-4791-b5fc-645b72981f53/volumes/kubernetes.io~projected/kube-api-access-ffmqk:{mountpoint:/var/lib/kubelet/pods/cb64abf9-519f-4791-b5fc-645b72981f53/volumes/kubernetes.io~projected/kube-api-access-ffmqk major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~projected/kube-api-access-7fszc:{mountpoint:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~projected/kube-api-access-7fszc major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8385728-e45b-41c9-b2cb-cce86024723b/volumes/kubernetes.io~projected/kube-api-access-xsnd2:{mountpoint:/var/lib/kubelet/pods/d8385728-e45b-41c9-b2cb-cce86024723b/volumes/kubernetes.io~projected/kube-api-access-xsnd2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~projected/kube-api-access-s78ql:{mountpoint:/var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~projected/kube-api-access-s78ql major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db09f543-253e-47ae-81bf-977c1e6323b7/volumes/kubernetes.io~projected/kube-api-access-5vd8b:{mountpoint:/var/lib/kubelet/pods/db09f543-253e-47ae-81bf-977c1e6323b7/volumes/kubernetes.io~projected/kube-api-access-5vd8b major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dca34023-3797-4998-9674-ee14daf13450/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dca34023-3797-4998-9674-ee14daf13450/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0bcf238-c8de-42ac-a905-348017d59ff8/volumes/kubernetes.io~projected/kube-api-access-m5j4v:{mountpoint:/var/lib/kubelet/pods/e0bcf238-c8de-42ac-a905-348017d59ff8/volumes/kubernetes.io~projected/kube-api-access-m5j4v major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f69fc23e-3928-4ff3-8c03-26c5b5aca388/volumes/kubernetes.io~projected/kube-api-access-cm68m:{mountpoint:/var/lib/kubelet/pods/f69fc23e-3928-4ff3-8c03-26c5b5aca388/volumes/kubernetes.io~projected/kube-api-access-cm68m major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/05c4474d7afb0111ff00e3944f4148dff0dd34f76450b71c526279abdf1bc68e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/501c443e8aa18c94dc59fc1d5d0f569b8938bf36a3c72fb76445c322b8e0314a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/22ce58b323742a99af19b97ae48dad34938499c0c8989145968e34fd6df213a4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2ab9bcc8b6fb51d2c1e287e713a6b3feb814cc59e5917be3e7f61e2e0ecf19b0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/75f8539492d7b9ba535b3651de4e7abcc6078ee7aa603df5087355f49973b295/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9397ef0aa817fd2baee958f314a03cad27cb7e462f73ff0a77ccebc401310ffa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bb65f1a1c0fb34e3520d0bc20973ca1f41498ce8ac4d4a099721aa3b5a10822c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3e10f60f3f2956e293822cafed07543298f41a723b41e936276c6db576fb88c2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/29eb509d04dd106834e884f719c445a8b52752b17d214d5c4530dcfe1719fa4e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7e49ce5503320fd56ded2d433c438f9180b16d6f69d8b91103abc08979ca1671/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/bea4e8cc965de63dde39f1083ee61cd42497a9ebaa1edf6b2218b967e796b0bd/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/9a5d909614a28c0f11acfc1a164381d994f6da909780f258f616752714154368/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0f8a89ebc5d4b19c12cf323cac8ef6496ee8e9f9e8e220b3b12d2afde2357477/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c947caf91e68c8c6764581c44ef4576e34d251d216c59bcd37b7e2ab2cec1b80/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/604970f8cb1754818ab2421cf9b8b7e5204d22f12c784c6466252ab96f6e6a50/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/32525ecb55a864aaaa5a9090b4b043eb7704a7b1ff5cbd21f42d722d94662134/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0a0d7b2d158c6a0767f5a12c7ca557e799da888b6bd726b493642a5486be5fa1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/16f415a5944aa42dcb442a9736129ef64d04faf13364433315ee166793068ca8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/219faf99dd16fa01ad681c02517a14cd81f6fb7fbad63d5da7e903a4efe041eb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a72a9ff1bc2240decc522fb636ed2acd566fe75517e5e8896ea67e095708292e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/166052f2292da229d9a9309d262f20ee3ab7bf97ede5d48898c37356eb5f6118/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/fff48298b263d0c03b5ed1b8d44ee216319f149be4b71af8f53556746191f1ce/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/d57afc2e0ed6c92c02fa3fb77d822d7aec2b16ae11006bf042d1a80137d8883a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/ccf7308149569a518edd35f3bc471b6dd8363c28f7be92d51e2e178209bc1fad/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/afb1e4854709be67fcb0dc38b440eee549ead802aa5915e9447ebcdf73d9ac45/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/4cf6f3957e105f9c642a1ba0f0ed4f2c9264178a5844e5866f35531eaaf4bfc0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f8cbb69491c1b7b149a38178b528da870c38a538392cb3de2abe6204e33452d2/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e6469511cfa94e0fc6e7bc9aa6eb6d11a32f41efe2e08b05cf77d5a759d3c0e2/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a9f873e15e61a667a90296a28d146344a55d2ef77cbd4cbb8ab84b736791fbc9/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f3536d2815fa637a74a79e4a0f88bf8e8f57db7b44cb0f02a0c6fb62fed17bbd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/f4ddc056b9631a6fd6324cfe5c8a393b47411ce2e21ed3a653393c778c5d1eb3/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/f8ab91fde3590b417a76b83addbdeba29d73dc0881f904284b4a67aecddf0c31/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/f4e488ec570e85af9e1503ee8c82ff784be74cfddd3dea46658c3d32da51f0fd/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/2ec028d92e1cd8d47e280e52814ac2c7d2bb6f36df37cfef46eeb7556db8d5d3/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/6e955e484f5200fc7bb604ee8fd8db52f8de94f1aa244aebd88e75c4b571b8c5/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ddbe83d4c06ca45bbc0137e137b56461c627b527e728fb962f8fd64a73bacf69/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/317300cabccb569aeaa1872b5ea2accfa1ea73101bd2f33f3234a55e7bd8a137/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d4cda48eba206b21823713667dda8b356dd0f784d57eb1dce9e383c11913c100/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/2fb870ab5af67d338c14f1300eadc311d7d5e467602c8033f9f5490a57803cc0/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/d0232adb0ec6a8ff3ec277b545522616d58db072beb70204ba2a4b305cc793dd/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/74ec97f89aa6dfc9d1fa7bcffea2b6bf0f465a7eb80b19cd34ffb98f2f3cf871/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/2ce3459e781ac7d3190ebc437ce14791dcd66dd7ed7bfbf6627cd86cb0162788/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/7001b9e694bff276d24f872546c44b81ab4baa3204f645b0591052a47f66a6d6/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/67cc88b605c681b35cc366403e28ee9a9aee57f11c08a82c717c3db6ced956de/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/6d21c2f5bf169e4b15c235fa246438f0ac426a2a41c78964ae33d969ee82f1e0/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/cfc00cc834a99d8d7c74382982c1ae7bb40d495139b05e392c992b00a9c30529/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/439b0f45919e8618f5eb7120a977d12b2ba075eee5b7816eaf05c22e681c6017/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/69b3c167103f75c8d858881a0d3bdfc97a82d45e2b5ba39ec1247200f49d7c96/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/43105abd0674824f3775f22acdcaafd4f5dae1017d8bbf975fd641c8b44ec9b8/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/5f339e2511024bf64ea0bfaee7566bbe01ee8697c0bdac040a555d1d8e0c1900/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/cf9a673ea9ef7e8ec97e71945f652ea897c80f8269ac97f6775a5670b824d820/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/3598d58d23729e74bdba0d7fc35791716c63f954bc62f4c0a27771c62cd84ed3/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/23528b2e8d3fd88c2d2619286403f22f4bceb1845395abf8554aa69624a0f925/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a9b824de071301619885eb5be9b337d50542d2d7499a2a64efe57e3faa1df457/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/93d271feabb668e35e037bcd15eb8591c75f93887a0b856a3535d09d488ce039/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7bc23d9c4318c79fee2741767150618bca3aa4f8a9ea4248e41b2743f9425ba6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a155a7758540f2d3f1560803a9e4d235998ba618ba77a4c4d56b4545e6e45468/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/d42c0fac599f9a35cacf985fc62aa5c67a40b1f6ea422ff8fda1aecf0a09f4c2/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fc0f0ff541b7a25df9c831b4d7e76705b2064879982ed38597cc74d58b8eeb3a/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/eeec7741e0cec9e47c75c5e748ffd1e61b04b15b5e680eec955e578e60ea7e10/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8c3549b20d4dce0014b4e05c77cb8d3368ab4403423162e0085187493057a613/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/553f378f2ba81edc902c1323603a07bae7908f9877978448c886d177e18513fb/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.249369 5024 manager.go:217] Machine: {Timestamp:2026-10-30 18:00:49.24890668 +0000 UTC m=+0.069618459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102527 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd55e87506f88ffcfc0170640e27f SystemUUID:ec2fd55e-8750-6f88-ffcf-c0170640e27f BootID:bc6edec1-a293-4972-b1e1-2f7f94bbe004 Filesystems:[{Device:/run/containers/storage/overlay-containers/ca2c46aacae4037b6622ca48ff3aeb52e27ea42a7698921120b9fc9d43db5c49/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/db09f543-253e-47ae-81bf-977c1e6323b7/volumes/kubernetes.io~projected/kube-api-access-5vd8b DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c12954fcb20d8121f2877d86e28ca99cffb69ef2bfe47eba3ae90d951df3d5a2/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b070856-f46c-4ebb-965c-d8aaacb18baf/volumes/kubernetes.io~projected/kube-api-access-v7hhd DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~projected/kube-api-access-dbkwc DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1566535e94f37245eb4ff32a1222d9919ad496832c2acd236f89afbecc537919/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/655bf5c4-4149-4e0e-b03b-10a41946b0f1/volumes/kubernetes.io~projected/kube-api-access-nsqxd DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/451e21d439551bd69724348f1d479d52488fdc049532fa3781a6abf7647b0a5c/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7dea29c6-0a9e-4f81-8237-f9199715657f/volumes/kubernetes.io~projected/kube-api-access-j7xbb DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc0875773df7464dc8bb94d1a9c753c35c5523caff21c5235500fbaa0f75352e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~projected/kube-api-access-npfv7 DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47a7c100-1107-4314-aa33-526e92b42c1f/volumes/kubernetes.io~projected/kube-api-access-rqf6r DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88b1723f-92ef-412a-a3ba-8742fa8b1131/volumes/kubernetes.io~projected/kube-api-access-wmmq9 DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2166c14d-2366-4925-9576-1fccab9df12c/volumes/kubernetes.io~projected/kube-api-access-fzx2r DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9aef38824a15e1c3f5234b0ca18a48d1957ba3b86624b2d116a86e67aa2c62c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf/volumes/kubernetes.io~projected/kube-api-access-b6mw8 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3208b565-97ac-4ed9-9d36-a289544c5fd1/volumes/kubernetes.io~projected/kube-api-access-bwq2b DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b440ae4e-411c-48fd-a8ed-45596e6b4147/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5084064f-9031-4a3c-b74f-46abf27982a3/volumes/kubernetes.io~projected/kube-api-access-wx5cm DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf5eb41066bcfaa0dfd2d159b66975824a9b94d94706b83bbc88a3fae5bbc6b8/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76cd0c09677ddeab12e30a606c5836ded0a8c0182b0371cc7b9c092257c95a88/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/198d5cb8787ccba5c6b4ba0c5c627a7854b13c8a3dbf8fcb5d8a1d21a461b31b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db20e492518676a2ea78cbd2bb37e60b97f5777573c796abe787570b3f0b11f4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbe6a56f9a9b1673aeda3ba3c4913e49f3283002d7bfd3b40be3ebf994088bd6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fe43c59-0934-4c18-84f1-381abdcf553d/volumes/kubernetes.io~projected/kube-api-access-5k227 DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cd9755409bd32ba6949dcd62cdd53155f4ed9dfec84003b02a8d10e22ebae58/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~projected/kube-api-access-xbksp DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~projected/kube-api-access-s78ql DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dca34023-3797-4998-9674-ee14daf13450/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~projected/kube-api-access-t4sqc DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0239d118-e691-4ac3-9e13-e1ced388a733/volumes/kubernetes.io~projected/kube-api-access-mdb7l DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/kube-api-access-55cjl DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~projected/kube-api-access-nnxdr DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/908d48a8d7412e8ad1ce496132da0ec5a51c7102e57de3c7cfe9848a33d978fc/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d5af711-aeb9-4984-b639-1d6b13db2909/volumes/kubernetes.io~projected/kube-api-access-9g8xv DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8385728-e45b-41c9-b2cb-cce86024723b/volumes/kubernetes.io~projected/kube-api-access-xsnd2 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75d95a5a-9823-4002-a14c-c1c42f2cdbe4/volumes/kubernetes.io~projected/kube-api-access-k9g69 DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a7517a84-07c0-4210-9303-ac839cdd188b/volumes/kubernetes.io~projected/kube-api-access-mw28h DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb64abf9-519f-4791-b5fc-645b72981f53/volumes/kubernetes.io~projected/kube-api-access-ffmqk DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47517669-cee7-4488-827e-b605f1cbfbc0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f69fc23e-3928-4ff3-8c03-26c5b5aca388/volumes/kubernetes.io~projected/kube-api-access-cm68m DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~projected/kube-api-access-7fszc DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d861baa6-45e0-44b6-9336-1ed846e00b49/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f14411e5f75bdc586ece7a5217b804801c8699f4ab36626117cadd9c336db1e4/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e57189f20a808dcb7cda5478676404ffd2ad68c2d9b6b159c9da8bdd37ed7d35/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b78ec231-1951-451f-8ec8-b70e20b98fc5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d13a2b8fad05bf837c07fce7cd1b4888727c8567efe0c6deb43ba94ebf70abe4/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a9b438f53c86acfc3d869dfabf2a181b90804ad4f4eb5677bfeafc77e7008a1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0bcf238-c8de-42ac-a905-348017d59ff8/volumes/kubernetes.io~projected/kube-api-access-m5j4v DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52511988ee22a11f593e0e82b46eaaa17fbf57b62ce424ece5e4b274a34adc09/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf63e9279c84007d29f5855575441d61f6bb2267a750ae30343ad4fb39931c5b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf8a5734502294d9c7ab49fc71de3c59ae0919004d5aa9a2b5fb494467fc4a59/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84ceb8a0665e02e25e6cf450bb7dc2d817b6ff9b3a97113952f588aa2b77b00e/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0/volumes/kubernetes.io~projected/kube-api-access-29qnt DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd/volumes/kubernetes.io~projected/kube-api-access-c4njc DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84757f5b4061c8bb5fe1621396ffe703073824f1e5c7b90515b97e377d157bed/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfe91e54-4f38-43ac-a520-2fc8e083bea9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dedee6e56a8e21b60bac8a883caeba965fc686a8209dc1681590c2d6722d4bd8/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03/volumes/kubernetes.io~projected/kube-api-access-82fn6 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f079577-e6f9-463f-9be1-a1d411696045/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc31f98387c98d9f303071cef60df45e3a5a90bd7030f96481dbdea66e58ce9c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46e82c61-c992-432e-b1ab-dffdaadccf70/volumes/kubernetes.io~projected/kube-api-access-bht72 DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/891a4e9b-107c-45f6-bef6-d825fe05e079/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1566535e94f3724 MacAddress:5a:82:bf:d9:0a:18 Speed:10000 Mtu:8901} {Name:198d5cb8787ccba MacAddress:16:9f:13:d5:0a:25 Speed:10000 Mtu:8901} {Name:4a9b438f53c86ac MacAddress:da:23:ae:95:30:e9 Speed:10000 Mtu:8901} {Name:4cd9755409bd32b MacAddress:ba:4d:3a:6c:75:55 Speed:10000 Mtu:8901} {Name:76cd0c09677ddea MacAddress:52:07:63:ca:07:41 Speed:10000 Mtu:8901} {Name:84757f5b4061c8b MacAddress:3e:36:89:27:08:4a Speed:10000 Mtu:8901} {Name:84ceb8a0665e02e MacAddress:d6:d6:04:aa:b4:cb Speed:10000 Mtu:8901} {Name:908d48a8d7412e8 MacAddress:1e:c0:31:11:6b:dd Speed:10000 Mtu:8901} {Name:a9aef38824a15e1 MacAddress:3e:7b:92:b8:98:86 Speed:10000 Mtu:8901} {Name:bf63e9279c84007 MacAddress:62:e8:0b:c9:12:e7 Speed:10000 Mtu:8901} {Name:bf8a5734502294d MacAddress:7a:9c:2a:a7:5a:69 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dc:8b:86:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:f9:6a:16:00:9d Speed:0 Mtu:8901} {Name:c12954fcb20d812 MacAddress:a2:cf:df:90:14:0e Speed:10000 Mtu:8901} {Name:ca2c46aacae4037 MacAddress:32:bf:5b:d0:0f:45 Speed:10000 Mtu:8901} {Name:cf5eb41066bcfaa MacAddress:0a:47:da:60:85:0d Speed:10000 Mtu:8901} {Name:d13a2b8fad05bf8 MacAddress:7a:1e:ae:35:8a:39 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dc:8b:86:55 Speed:0 Mtu:9001} {Name:f14411e5f75bdc5 MacAddress:c6:1a:12:90:09:67 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:78:ff:bb:ce:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.250002 5024 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.250083 5024 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.250510 5024 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.250566 5024 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251043 5024 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251056 5024 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251074 5024 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251100 5024 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:49.251427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251128 5024 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:00:49.252079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.251484 5024 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:00:49.252079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252000 5024 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:00:49.252079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252057 5024 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:00:49.252224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252086 5024 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:00:49.252224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252104 5024 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:00:49.252224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252117 5024 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:00:49.252224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.252129 5024 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:00:49.253071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.253052 5024 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:00:49.253071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.253069 5024 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:49.253671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.253653 5024 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:00:49.255807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.255793 5024 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:00:49.255807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.255807 5024 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:49.255904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.255894 5024 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:00:49.256087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256077 5024 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:00:49.256121 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256106 5024 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:00:49.256208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256199 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256215 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256221 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256226 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256231 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256237 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:00:49.256243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256243 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256248 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256255 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256260 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256268 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256277 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256304 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:00:49.256443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.256310 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:00:49.259846 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.259831 5024 server.go:1263] "Started kubelet" Oct 30 18:00:49.260054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.259965 5024 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:00:49.260151 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.259933 5024 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:00:49.260209 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.260180 5024 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:00:49.260434 ip-10-0-143-94 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:00:49.261463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.260808 5024 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:00:49.263577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263505 5024 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:00:49.263577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263524 5024 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:49.263762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263582 5024 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:00:49.263762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263590 5024 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:00:49.263762 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.263643 5024 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-94.ec2.internal\" not found" Oct 30 18:00:49.263762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263571 5024 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:09 +0000 UTC" deadline="2028-03-28 01:40:34.893363708 +0000 UTC" Oct 30 18:00:49.263762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.263702 5024 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12343h39m45.629665146s" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.267890 5024 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268040 5024 factory.go:55] Registering systemd factory Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268049 5024 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268288 5024 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268469 5024 factory.go:153] Registering CRI-O factory Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268484 5024 factory.go:223] Registration of the crio container factory successfully Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268504 5024 factory.go:103] Registering Raw factory Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.268519 5024 manager.go:1199] Started watching for new ooms in manager Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269189 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69fc23e-3928-4ff3-8c03-26c5b5aca388" volumeName="kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269238 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269280 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46e82c61-c992-432e-b1ab-dffdaadccf70" volumeName="kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269294 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d95a5a-9823-4002-a14c-c1c42f2cdbe4" volumeName="kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269308 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dea29c6-0a9e-4f81-8237-f9199715657f" volumeName="kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269326 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb64abf9-519f-4791-b5fc-645b72981f53" volumeName="kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269339 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe91e54-4f38-43ac-a520-2fc8e083bea9" volumeName="kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269393 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8385728-e45b-41c9-b2cb-cce86024723b" volumeName="kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269410 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d95a5a-9823-4002-a14c-c1c42f2cdbe4" volumeName="kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269422 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dea29c6-0a9e-4f81-8237-f9199715657f" volumeName="kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269500 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b1723f-92ef-412a-a3ba-8742fa8b1131" volumeName="kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269514 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3208b565-97ac-4ed9-9d36-a289544c5fd1" volumeName="kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269533 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3208b565-97ac-4ed9-9d36-a289544c5fd1" volumeName="kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269545 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269655 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47a7c100-1107-4314-aa33-526e92b42c1f" volumeName="kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269669 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7517a84-07c0-4210-9303-ac839cdd188b" volumeName="kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269688 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db09f543-253e-47ae-81bf-977c1e6323b7" volumeName="kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269700 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712baeaa-edb0-45ce-bcd4-9a29c7a5eec0" volumeName="kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269719 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d95a5a-9823-4002-a14c-c1c42f2cdbe4" volumeName="kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269737 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5af711-aeb9-4984-b639-1d6b13db2909" volumeName="kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269782 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfe69b0-a077-4fd0-9f7e-a513c1b882bf" volumeName="kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269801 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fe43c59-0934-4c18-84f1-381abdcf553d" volumeName="kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269825 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78ec231-1951-451f-8ec8-b70e20b98fc5" volumeName="kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269838 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78ec231-1951-451f-8ec8-b70e20b98fc5" volumeName="kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269855 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269868 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712baeaa-edb0-45ce-bcd4-9a29c7a5eec0" volumeName="kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.269993 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe91e54-4f38-43ac-a520-2fc8e083bea9" volumeName="kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.270010 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8385728-e45b-41c9-b2cb-cce86024723b" volumeName="kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config" seLinuxMountContext="" Oct 30 18:00:49.273123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.273034 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861baa6-45e0-44b6-9336-1ed846e00b49" volumeName="kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca" seLinuxMountContext="" Oct 30 18:00:49.275707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.275687 5024 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.273075 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dca34023-3797-4998-9674-ee14daf13450" volumeName="kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276560 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f69fc23e-3928-4ff3-8c03-26c5b5aca388" volumeName="kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276578 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276610 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b070856-f46c-4ebb-965c-d8aaacb18baf" volumeName="kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276623 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276636 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16a01d63-08e8-4859-95c8-b89108043583" volumeName="kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276649 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd" volumeName="kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config" seLinuxMountContext="" Oct 30 18:00:49.276937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276660 5024 manager.go:319] Starting recovery of all containers Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.276663 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277733 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46e82c61-c992-432e-b1ab-dffdaadccf70" volumeName="kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277756 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712baeaa-edb0-45ce-bcd4-9a29c7a5eec0" volumeName="kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277776 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47517669-cee7-4488-827e-b605f1cbfbc0" volumeName="kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277789 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5084064f-9031-4a3c-b74f-46abf27982a3" volumeName="kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277802 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dea29c6-0a9e-4f81-8237-f9199715657f" volumeName="kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277833 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="891a4e9b-107c-45f6-bef6-d825fe05e079" volumeName="kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277847 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fe43c59-0934-4c18-84f1-381abdcf553d" volumeName="kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277860 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861baa6-45e0-44b6-9336-1ed846e00b49" volumeName="kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277875 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db09f543-253e-47ae-81bf-977c1e6323b7" volumeName="kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277888 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277903 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277916 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46e82c61-c992-432e-b1ab-dffdaadccf70" volumeName="kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277935 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47517669-cee7-4488-827e-b605f1cbfbc0" volumeName="kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277949 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8385728-e45b-41c9-b2cb-cce86024723b" volumeName="kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277966 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861baa6-45e0-44b6-9336-1ed846e00b49" volumeName="kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277980 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0239d118-e691-4ac3-9e13-e1ced388a733" volumeName="kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.277993 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d5af711-aeb9-4984-b639-1d6b13db2909" volumeName="kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278042 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278058 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278085 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278099 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b440ae4e-411c-48fd-a8ed-45596e6b4147" volumeName="kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278112 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62cbcc9b-a1ea-4304-9fe7-1805deaf6f03" volumeName="kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278127 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dea29c6-0a9e-4f81-8237-f9199715657f" volumeName="kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278142 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278155 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="891a4e9b-107c-45f6-bef6-d825fe05e079" volumeName="kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278170 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe91e54-4f38-43ac-a520-2fc8e083bea9" volumeName="kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278183 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861baa6-45e0-44b6-9336-1ed846e00b49" volumeName="kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278196 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712baeaa-edb0-45ce-bcd4-9a29c7a5eec0" volumeName="kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278209 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278222 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0bcf238-c8de-42ac-a905-348017d59ff8" volumeName="kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278255 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278270 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="655bf5c4-4149-4e0e-b03b-10a41946b0f1" volumeName="kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278284 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712baeaa-edb0-45ce-bcd4-9a29c7a5eec0" volumeName="kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278301 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8528ac59-25e8-435f-9d93-599e19af51c5" volumeName="kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278313 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b440ae4e-411c-48fd-a8ed-45596e6b4147" volumeName="kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278327 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db09f543-253e-47ae-81bf-977c1e6323b7" volumeName="kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278340 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278355 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2166c14d-2366-4925-9576-1fccab9df12c" volumeName="kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278369 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88b1723f-92ef-412a-a3ba-8742fa8b1131" volumeName="kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278383 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="891a4e9b-107c-45f6-bef6-d825fe05e079" volumeName="kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278397 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b78ec231-1951-451f-8ec8-b70e20b98fc5" volumeName="kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278448 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278464 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd" volumeName="kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278477 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3208b565-97ac-4ed9-9d36-a289544c5fd1" volumeName="kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278503 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278511 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cfe69b0-a077-4fd0-9f7e-a513c1b882bf" volumeName="kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278518 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db09f543-253e-47ae-81bf-977c1e6323b7" volumeName="kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278534 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f079577-e6f9-463f-9be1-a1d411696045" volumeName="kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278565 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f409e7d-56d0-43ed-ac57-5724283f1a57" volumeName="kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets" seLinuxMountContext="" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278581 5024 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:00:49.278625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.278587 5024 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:00:49.287917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.287893 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:49.288048 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.287938 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:49.295523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.295499 5024 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:00:49.297461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.297438 5024 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:00:49.297461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.297462 5024 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:00:49.297662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.297483 5024 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:00:49.297662 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.297518 5024 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:00:49.311179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.311153 5024 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:00:49.311336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.311318 5024 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:49.315182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.315158 5024 generic.go:356] "Generic (PLEG): container finished" podID="e5baff094b56b89bb07e9432557fee4f" containerID="47fa4420b4cc6b164de9d46b9c8e441f6f0583d1f1f94f73b0fdd59452c31fde" exitCode=0 Oct 30 18:00:49.322752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.322725 5024 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:49.323012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.322994 5024 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:49.323223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.323197 5024 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348082 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="3f94680a9db51268516226ae389c0e7e926987c663d3d90079c4eb9bfbc627d0" exitCode=0 Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348114 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="8b35a5e5b804874accdbf62bcd8bf370ccf2b8b0a803fef79da01547a26acf36" exitCode=0 Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348126 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="ee01ec95acfeebf9566d72903fd1f500936e0d5ff92f75366f712d246274a3d6" exitCode=0 Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348155 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="ad322a50215855ac5e212ce1db47469d7bd1d94364a1226806f6855b7bb83af5" exitCode=0 Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348166 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="d262cc914c6d76f7a8933e6bbc7b7c8b765e6e2c96cbea7a16b46394cfd230c2" exitCode=0 Oct 30 18:00:49.348425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.348176 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dea29c6-0a9e-4f81-8237-f9199715657f" containerID="e2739eee1dd8812468dcb479e3d306419bb4efe3635912334522b08b6da33c2c" exitCode=0 Oct 30 18:00:49.355185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.355128 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:49.355185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.355156 5024 generic.go:356] "Generic (PLEG): container finished" podID="d861baa6-45e0-44b6-9336-1ed846e00b49" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" exitCode=255 Oct 30 18:00:49.377226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.377206 5024 manager.go:324] Recovery completed Oct 30 18:00:49.398325 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.398297 5024 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 18:00:49.398958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.398943 5024 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:00:49.398958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.398957 5024 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:00:49.399034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.398975 5024 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:00:49.399100 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399087 5024 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:00:49.399132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399101 5024 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:00:49.399132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399120 5024 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:00:49.399132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399125 5024 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:00:49.399132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399131 5024 policy_none.go:50] "Start" Oct 30 18:00:49.399238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399146 5024 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:00:49.399238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399155 5024 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:00:49.399289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399275 5024 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:00:49.399289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399282 5024 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:00:49.399289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.399287 5024 policy_none.go:44] "Start" Oct 30 18:00:49.403662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.403632 5024 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:00:49.407142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.407128 5024 manager.go:341] "Starting Device Plugin manager" Oct 30 18:00:49.407879 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.407168 5024 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:00:49.407936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.407887 5024 server.go:87] "Starting device plugin registration server" Oct 30 18:00:49.408238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408225 5024 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:00:49.408271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408240 5024 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:00:49.408342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408324 5024 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:00:49.408453 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408424 5024 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:00:49.408498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408463 5024 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:00:49.408498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408469 5024 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:00:49.408609 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.408522 5024 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:00:49.408451308Z","UUID":"3a6e107b-319a-4ae9-9be5-1c09ef9c0a0d","Handler":null,"Name":"","Endpoint":""} Oct 30 18:00:49.409770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.409748 5024 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:00:49.409847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.409789 5024 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:00:49.410078 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.410055 5024 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:00:49.508783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.508754 5024 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:49.510076 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.510052 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:49.510188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.510085 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:49.510188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.510100 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-94.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:49.510188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.510144 5024 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.538641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.538616 5024 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.538745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.538683 5024 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.598611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.598522 5024 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal"] Oct 30 18:00:49.599179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599149 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerStarted","Data":"edb00c8462801837e811135b920422d1d7dc112717892e6abd010a87a822d2e2"} Oct 30 18:00:49.599216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599186 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerDied","Data":"47fa4420b4cc6b164de9d46b9c8e441f6f0583d1f1f94f73b0fdd59452c31fde"} Oct 30 18:00:49.599216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599196 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.599292 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599202 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" event={"ID":"e5baff094b56b89bb07e9432557fee4f","Type":"ContainerStarted","Data":"9680290ed7ecdfbf514a5095cf694d822d23883c808f8bfc7c199239e5249ef0"} Oct 30 18:00:49.599343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599322 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" event={"ID":"00fcd38864cc26968aedfdf83ce5b096","Type":"ContainerStarted","Data":"757e506de47e59c0394b0ac21822f457395771318431becd6041d500610f1106"} Oct 30 18:00:49.599373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599355 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" event={"ID":"00fcd38864cc26968aedfdf83ce5b096","Type":"ContainerStarted","Data":"6c12c27a5cb984eb4f5b22758151fafd29baa0ef5a197a74ffb6904c84e13493"} Oct 30 18:00:49.599406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.599382 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.623402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.623352 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:49.623402 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.623393 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.623696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.623684 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:49.623736 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:49.623715 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.713030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.712990 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.713171 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.713042 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.713171 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.713061 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813569 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813628 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813660 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813676 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813713 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/00fcd38864cc26968aedfdf83ce5b096-config\") pod \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" (UID: \"00fcd38864cc26968aedfdf83ce5b096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:49.813884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:49.813751 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5baff094b56b89bb07e9432557fee4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" (UID: \"e5baff094b56b89bb07e9432557fee4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:50.253321 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.253283 5024 apiserver.go:52] "Watching apiserver" Oct 30 18:00:50.296976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.296943 5024 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:00:50.297666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.297636 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-79lsr","openshift-service-ca/service-ca-5fffc97bdf-6qxt6","openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx","openshift-insights/periodic-gathering-kfb2l-ngk6j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w","openshift-multus/multus-additional-cni-plugins-4wtdk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd","openshift-image-registry/image-registry-6f75c69b8c-gmvng","openshift-console-operator/console-operator-7cd56b8dbc-ts48c","openshift-network-console/networking-console-plugin-86c478bb88-pllqc","openshift-multus/multus-xb89g","openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn","kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal","kube-system/kas-connection-checker-685895c7c9-ztx4c","openshift-ingress-canary/ingress-canary-dq8ks","kube-system/kas-connection-checker-685895c7c9-lfmlk","kube-system/konnectivity-agent-dw7mk","openshift-dns/dns-default-fnrlr","openshift-image-registry/node-ca-rzdgf","openshift-insights/insights-operator-7c4986d449-kzr8f","openshift-ovn-kubernetes/ovnkube-node-lkmzz","kube-system/kas-connection-checker-685895c7c9-l6m5w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd","kube-system/global-pull-secret-syncer-jzwlg","openshift-ingress/router-default-7765667bbf-4zpgd","openshift-insights/insights-runtime-extractor-7bvmg","openshift-network-diagnostics/network-check-target-2tn9m","openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9","openshift-cluster-node-tuning-operator/tuned-rz2lx","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6","openshift-network-operator/iptables-alerter-jk8c2","openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l","openshift-image-registry/image-registry-85bf557f7d-4tfww","openshift-dns/node-resolver-f72ft","openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj"] Oct 30 18:00:50.303162 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.303139 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.303784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.303767 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.304163 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.304140 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.305292 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.305273 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.305815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.305790 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:50.309399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.309381 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.310926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.310906 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.311011 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.310955 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.311984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.311969 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.313334 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.313317 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.314174 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.314158 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.314341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.314324 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.314765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.314743 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.314843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.314829 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6wcdg\"" Oct 30 18:00:50.315193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.315172 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.315426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.315401 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 18:00:50.315512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.315489 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 18:00:50.315717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.315695 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wvmzn\"" Oct 30 18:00:50.315926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.315911 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.316108 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.316091 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 18:00:50.316243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.316225 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 18:00:50.317491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.317473 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.317692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.317676 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.318136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.318119 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 18:00:50.318305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.318290 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 18:00:50.318402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.318290 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cbhh9\"" Oct 30 18:00:50.320098 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.320069 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 18:00:50.320170 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.320103 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.320378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.320363 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.320495 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.320417 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.322358 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.322336 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:00:50.322470 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.322455 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 18:00:50.322840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.322823 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:00:50.323772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.323756 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 18:00:50.324014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.323985 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v5bj5\"" Oct 30 18:00:50.324014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324006 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.324273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324257 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.324423 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324273 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.324423 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324305 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 18:00:50.324570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324487 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:00:50.324570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324306 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:00:50.324570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324409 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.324570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324307 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.324768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324672 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 18:00:50.324768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324736 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.324768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324763 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.324901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324767 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kkf44\"" Oct 30 18:00:50.324951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324898 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 18:00:50.324951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324945 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.325047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324990 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.325047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.324940 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kr5jk\"" Oct 30 18:00:50.325047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325026 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.325186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325151 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:00:50.325186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325159 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8sxnb\"" Oct 30 18:00:50.325288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325187 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.325288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325196 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:00:50.325288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325239 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:00:50.325288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325264 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:00:50.325288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325274 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:00:50.325528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325305 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 18:00:50.325528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325314 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 18:00:50.325528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325407 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:00:50.325528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325414 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 18:00:50.325528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325451 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325560 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325579 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325582 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nsbpz\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325646 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325679 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.325780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325701 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:00:50.325973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.325795 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.328282 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.328260 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qhhbf\"" Oct 30 18:00:50.328282 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.328266 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg98\"" Oct 30 18:00:50.328414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.328399 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:00:50.328511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.328440 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-shsdq\"" Oct 30 18:00:50.330335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.330320 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:00:50.332563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.332549 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:00:50.334578 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.334564 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 18:00:50.336046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336019 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:00:50.336252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336226 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:00:50.336446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336426 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:00:50.336529 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336482 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:00:50.336585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336531 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:00:50.336585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336575 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:00:50.336720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336659 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:00:50.336720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336673 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j6bp8\"" Oct 30 18:00:50.336720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336697 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-njkgb\"" Oct 30 18:00:50.336855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336747 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:00:50.336855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336776 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:00:50.336855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336833 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:00:50.337001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336903 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:00:50.337001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.336984 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:00:50.337099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.337038 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:00:50.337099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.337073 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:00:50.337224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.337102 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:00:50.337224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.337156 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:00:50.337224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.337195 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:00:50.338032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.338008 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 18:00:50.338250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.338237 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:00:50.338310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.338295 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:00:50.367409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.367392 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:50.367531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.367518 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:50.371914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.371882 5024 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:00:50.388136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.388115 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:50.388226 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.388161 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-94.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-94.ec2.internal" Oct 30 18:00:50.388570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.388556 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:50.388633 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.388591 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-94.ec2.internal" Oct 30 18:00:50.411650 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411627 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411636 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411685 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411687 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411735 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411773 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411777 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411786 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411795 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411799 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411819 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411800 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411836 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411843 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411850 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411851 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411865 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411871 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411858 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411785 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411893 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411906 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:00:50.411966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.411969 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:00:50.417205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417189 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.417260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417213 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.417260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417228 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.417260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417246 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:50.417385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417295 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.417385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417319 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.417385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j4v\" (UniqueName: \"kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v\") pod \"migrator-66df5d496b-mzqd9\" (UID: \"e0bcf238-c8de-42ac-a905-348017d59ff8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417424 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88b1723f-92ef-412a-a3ba-8742fa8b1131-data-volume\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417453 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417479 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417491 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-snapshots\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.417535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417501 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417541 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417585 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417646 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417651 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/891a4e9b-107c-45f6-bef6-d825fe05e079-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417682 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdb7l\" (UniqueName: \"kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l\") pod \"network-check-source-6f78d59f99-7ncwj\" (UID: \"0239d118-e691-4ac3-9e13-e1ced388a733\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417726 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mw8\" (UniqueName: \"kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5cm\" (UniqueName: \"kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm\") pod \"kas-connection-checker-685895c7c9-ztx4c\" (UID: \"5084064f-9031-4a3c-b74f-46abf27982a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417772 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417797 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.417829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417823 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417837 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b440ae4e-411c-48fd-a8ed-45596e6b4147-konnectivity-ca\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417846 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417951 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.417996 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418040 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418078 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418136 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418182 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-script-lib\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.418195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418191 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmqk\" (UniqueName: \"kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk\") pod \"migrator-66df5d496b-v8jsn\" (UID: \"cb64abf9-519f-4791-b5fc-645b72981f53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418223 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418240 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418255 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418281 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418311 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hhd\" (UniqueName: \"kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd\") pod \"kas-connection-checker-685895c7c9-l6m5w\" (UID: \"3b070856-f46c-4ebb-965c-d8aaacb18baf\") " pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418328 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418344 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx2r\" (UniqueName: \"kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418362 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418386 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq2b\" (UniqueName: \"kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418430 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16a01d63-08e8-4859-95c8-b89108043583-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:50.418458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418438 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418482 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418524 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82fn6\" (UniqueName: \"kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6\") pod \"volume-data-source-validator-58c9f5f46d-ds2qd\" (UID: \"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418620 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418681 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0f079577-e6f9-463f-9be1-a1d411696045-serving-cert\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418711 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418739 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418766 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbksp\" (UniqueName: \"kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418793 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418825 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418852 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.418904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418890 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418912 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b440ae4e-411c-48fd-a8ed-45596e6b4147-agent-certs\") pod \"konnectivity-agent-dw7mk\" (UID: \"b440ae4e-411c-48fd-a8ed-45596e6b4147\") " pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.418925 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g8xv\" (UniqueName: \"kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419023 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-cni-binary-copy\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419040 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-trusted-ca\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419053 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dca34023-3797-4998-9674-ee14daf13450-original-pull-secret\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419066 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419111 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419178 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419209 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419183 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419236 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bht72\" (UniqueName: \"kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419244 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8385728-e45b-41c9-b2cb-cce86024723b-multus-daemon-config\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419264 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419289 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419316 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnd2\" (UniqueName: \"kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.419434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419333 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d861baa6-45e0-44b6-9336-1ed846e00b49-serving-cert\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419444 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7d5af711-aeb9-4984-b639-1d6b13db2909-serviceca\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419496 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovn-node-metrics-cert\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419502 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419546 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419574 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419649 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419674 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419708 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419783 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419812 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419838 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419873 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419875 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d861baa6-45e0-44b6-9336-1ed846e00b49-config\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419891 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419937 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-service-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.419982 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420028 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420090 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420126 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-default-certificate\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.420203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420195 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420219 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420237 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420244 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-ovnkube-config\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420304 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420306 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-tuned\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420338 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420379 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420401 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420439 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420516 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4njc\" (UniqueName: \"kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420538 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420546 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-key\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420577 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420623 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxdr\" (UniqueName: \"kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420643 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420660 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420683 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46e82c61-c992-432e-b1ab-dffdaadccf70-serving-cert\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420681 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420729 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420749 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420775 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47517669-cee7-4488-827e-b605f1cbfbc0-cert\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420805 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420897 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420915 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-signing-cabundle\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420952 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k227\" (UniqueName: \"kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.420951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420958 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9fe43c59-0934-4c18-84f1-381abdcf553d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.420988 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421048 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421087 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421106 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421122 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421138 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421153 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421171 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421202 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421238 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421264 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421294 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421311 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421341 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46e82c61-c992-432e-b1ab-dffdaadccf70-config\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421344 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f69fc23e-3928-4ff3-8c03-26c5b5aca388-iptables-alerter-script\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421418 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421430 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-env-overrides\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421463 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421578 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421624 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421642 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9g69\" (UniqueName: \"kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421657 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421677 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421720 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fszc\" (UniqueName: \"kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421729 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421754 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421771 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421781 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421827 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421863 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421903 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421930 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421958 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qnt\" (UniqueName: \"kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421978 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.421982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.421989 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-stats-auth\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422036 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422057 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422074 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422098 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422115 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78ql\" (UniqueName: \"kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422131 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422146 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422179 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422271 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422358 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422395 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422421 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422447 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmq9\" (UniqueName: \"kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422451 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f079577-e6f9-463f-9be1-a1d411696045-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-tmp-dir\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422633 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xbb\" (UniqueName: \"kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422662 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkwc\" (UniqueName: \"kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422689 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422716 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqf6r\" (UniqueName: \"kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r\") pod \"kas-connection-checker-685895c7c9-lfmlk\" (UID: \"47a7c100-1107-4314-aa33-526e92b42c1f\") " pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422743 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422767 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422794 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sqc\" (UniqueName: \"kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422837 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422888 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422930 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.422988 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423011 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423090 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423108 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423131 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423161 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423182 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423245 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423268 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe91e54-4f38-43ac-a520-2fc8e083bea9-tmp\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423273 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3208b565-97ac-4ed9-9d36-a289544c5fd1-tmp-dir\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423288 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3208b565-97ac-4ed9-9d36-a289544c5fd1-config-volume\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423310 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423332 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f079577-e6f9-463f-9be1-a1d411696045-tmp\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423357 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/891a4e9b-107c-45f6-bef6-d825fe05e079-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423356 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423397 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423422 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqxd\" (UniqueName: \"kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423448 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm68m\" (UniqueName: \"kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423587 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.423769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423706 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dea29c6-0a9e-4f81-8237-f9199715657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.424330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.423821 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.436804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.436779 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hhd\" (UniqueName: \"kubernetes.io/projected/3b070856-f46c-4ebb-965c-d8aaacb18baf-kube-api-access-v7hhd\") pod \"kas-connection-checker-685895c7c9-l6m5w\" (UID: \"3b070856-f46c-4ebb-965c-d8aaacb18baf\") " pod="kube-system/kas-connection-checker-685895c7c9-l6m5w" Oct 30 18:00:50.436952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.436838 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq2b\" (UniqueName: \"kubernetes.io/projected/3208b565-97ac-4ed9-9d36-a289544c5fd1-kube-api-access-bwq2b\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.436952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.436893 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5cm\" (UniqueName: \"kubernetes.io/projected/5084064f-9031-4a3c-b74f-46abf27982a3-kube-api-access-wx5cm\") pod \"kas-connection-checker-685895c7c9-ztx4c\" (UID: \"5084064f-9031-4a3c-b74f-46abf27982a3\") " pod="kube-system/kas-connection-checker-685895c7c9-ztx4c" Oct 30 18:00:50.436952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.436935 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j4v\" (UniqueName: \"kubernetes.io/projected/e0bcf238-c8de-42ac-a905-348017d59ff8-kube-api-access-m5j4v\") pod \"migrator-66df5d496b-mzqd9\" (UID: \"e0bcf238-c8de-42ac-a905-348017d59ff8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mzqd9" Oct 30 18:00:50.437094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.437022 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmqk\" (UniqueName: \"kubernetes.io/projected/cb64abf9-519f-4791-b5fc-645b72981f53-kube-api-access-ffmqk\") pod \"migrator-66df5d496b-v8jsn\" (UID: \"cb64abf9-519f-4791-b5fc-645b72981f53\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-v8jsn" Oct 30 18:00:50.437094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.437053 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdb7l\" (UniqueName: \"kubernetes.io/projected/0239d118-e691-4ac3-9e13-e1ced388a733-kube-api-access-mdb7l\") pod \"network-check-source-6f78d59f99-7ncwj\" (UID: \"0239d118-e691-4ac3-9e13-e1ced388a733\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7ncwj" Oct 30 18:00:50.437289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.437271 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mw8\" (UniqueName: \"kubernetes.io/projected/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-kube-api-access-b6mw8\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.438086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.438069 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx2r\" (UniqueName: \"kubernetes.io/projected/2166c14d-2366-4925-9576-1fccab9df12c-kube-api-access-fzx2r\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.439021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.439006 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbksp\" (UniqueName: \"kubernetes.io/projected/b78ec231-1951-451f-8ec8-b70e20b98fc5-kube-api-access-xbksp\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.439849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.439833 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.440071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.440056 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82fn6\" (UniqueName: \"kubernetes.io/projected/62cbcc9b-a1ea-4304-9fe7-1805deaf6f03-kube-api-access-82fn6\") pod \"volume-data-source-validator-58c9f5f46d-ds2qd\" (UID: \"62cbcc9b-a1ea-4304-9fe7-1805deaf6f03\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ds2qd" Oct 30 18:00:50.441831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.441788 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g8xv\" (UniqueName: \"kubernetes.io/projected/7d5af711-aeb9-4984-b639-1d6b13db2909-kube-api-access-9g8xv\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.441941 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.441926 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bht72\" (UniqueName: \"kubernetes.io/projected/46e82c61-c992-432e-b1ab-dffdaadccf70-kube-api-access-bht72\") pod \"service-ca-operator-544d749c8c-xzv5l\" (UID: \"46e82c61-c992-432e-b1ab-dffdaadccf70\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" Oct 30 18:00:50.442094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.442079 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw28h\" (UniqueName: \"kubernetes.io/projected/a7517a84-07c0-4210-9303-ac839cdd188b-kube-api-access-mw28h\") pod \"network-check-target-2tn9m\" (UID: \"a7517a84-07c0-4210-9303-ac839cdd188b\") " pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:50.442163 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.442148 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnd2\" (UniqueName: \"kubernetes.io/projected/d8385728-e45b-41c9-b2cb-cce86024723b-kube-api-access-xsnd2\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.442934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.442915 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k227\" (UniqueName: \"kubernetes.io/projected/9fe43c59-0934-4c18-84f1-381abdcf553d-kube-api-access-5k227\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.443020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.442981 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.443109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.443090 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.443222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.443172 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sqc\" (UniqueName: \"kubernetes.io/projected/0f079577-e6f9-463f-9be1-a1d411696045-kube-api-access-t4sqc\") pod \"insights-operator-7c4986d449-kzr8f\" (UID: \"0f079577-e6f9-463f-9be1-a1d411696045\") " pod="openshift-insights/insights-operator-7c4986d449-kzr8f" Oct 30 18:00:50.443339 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.443321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmq9\" (UniqueName: \"kubernetes.io/projected/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-api-access-wmmq9\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.444537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.444520 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78ql\" (UniqueName: \"kubernetes.io/projected/d861baa6-45e0-44b6-9336-1ed846e00b49-kube-api-access-s78ql\") pod \"console-operator-7cd56b8dbc-ts48c\" (UID: \"d861baa6-45e0-44b6-9336-1ed846e00b49\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:50.446021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446005 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xbb\" (UniqueName: \"kubernetes.io/projected/7dea29c6-0a9e-4f81-8237-f9199715657f-kube-api-access-j7xbb\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446448 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxdr\" (UniqueName: \"kubernetes.io/projected/47517669-cee7-4488-827e-b605f1cbfbc0-kube-api-access-nnxdr\") pod \"ingress-canary-dq8ks\" (UID: \"47517669-cee7-4488-827e-b605f1cbfbc0\") " pod="openshift-ingress-canary/ingress-canary-dq8ks" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446673 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqxd\" (UniqueName: \"kubernetes.io/projected/655bf5c4-4149-4e0e-b03b-10a41946b0f1-kube-api-access-nsqxd\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446791 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkwc\" (UniqueName: \"kubernetes.io/projected/891a4e9b-107c-45f6-bef6-d825fe05e079-kube-api-access-dbkwc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w\" (UID: \"891a4e9b-107c-45f6-bef6-d825fe05e079\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"periodic-gathering-kfb2l-ngk6j\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446930 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fszc\" (UniqueName: \"kubernetes.io/projected/cfe91e54-4f38-43ac-a520-2fc8e083bea9-kube-api-access-7fszc\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446943 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9g69\" (UniqueName: \"kubernetes.io/projected/75d95a5a-9823-4002-a14c-c1c42f2cdbe4-kube-api-access-k9g69\") pod \"service-ca-5fffc97bdf-6qxt6\" (UID: \"75d95a5a-9823-4002-a14c-c1c42f2cdbe4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6qxt6" Oct 30 18:00:50.446970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446968 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.447212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446993 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqf6r\" (UniqueName: \"kubernetes.io/projected/47a7c100-1107-4314-aa33-526e92b42c1f-kube-api-access-rqf6r\") pod \"kas-connection-checker-685895c7c9-lfmlk\" (UID: \"47a7c100-1107-4314-aa33-526e92b42c1f\") " pod="kube-system/kas-connection-checker-685895c7c9-lfmlk" Oct 30 18:00:50.447212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.446965 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qnt\" (UniqueName: \"kubernetes.io/projected/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-kube-api-access-29qnt\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.447212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.447092 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm68m\" (UniqueName: \"kubernetes.io/projected/f69fc23e-3928-4ff3-8c03-26c5b5aca388-kube-api-access-cm68m\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.448099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.448083 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4njc\" (UniqueName: \"kubernetes.io/projected/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-kube-api-access-c4njc\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.524289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524213 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.524289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524253 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.524289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524324 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524350 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524373 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-var-lib-kubelet\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524377 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524417 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524420 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524428 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-kubelet\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524451 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524470 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-kubelet\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524495 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524502 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-system-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524451 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-var-lib-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524520 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524521 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-socket-dir-parent\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524563 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524572 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-cni-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.524619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524625 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524643 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-bin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524658 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524667 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524669 5024 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524713 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524736 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524750 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-lib-modules\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524758 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524789 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-slash\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524825 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.524937 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-socket-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.525412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525412 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525441 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525444 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysconfig\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525469 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525776 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525799 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525834 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525864 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525885 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525910 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525934 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525959 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525986 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.526010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526009 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526034 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526063 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526088 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526112 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526141 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526176 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526229 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526254 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526310 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526338 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526373 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526399 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526425 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526447 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526473 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526499 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526525 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526586 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526630 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526660 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526693 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.526704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526697 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d5af711-aeb9-4984-b639-1d6b13db2909-host\") pod \"node-ca-rzdgf\" (UID: \"7d5af711-aeb9-4984-b639-1d6b13db2909\") " pod="openshift-image-registry/node-ca-rzdgf" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526720 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525524 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-kubelet-config\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526772 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526780 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-systemd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.525682 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-sysctl-conf\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.526795 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527309 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-run\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527358 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527383 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527406 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527416 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-etc-kubernetes\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527442 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527458 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-netns\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527472 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527526 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"image-registry-85bf557f7d-4tfww\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527581 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-registration-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527583 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-sys\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527642 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-etc-openvswitch\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527662 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8cfe69b0-a077-4fd0-9f7e-a513c1b882bf-hosts-file\") pod \"node-resolver-f72ft\" (UID: \"8cfe69b0-a077-4fd0-9f7e-a513c1b882bf\") " pod="openshift-dns/node-resolver-f72ft" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-system-cni-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527707 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-host\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527728 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88b1723f-92ef-412a-a3ba-8742fa8b1131-crio-socket\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527752 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-cnibin\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527763 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-var-lib-cni-multus\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527778 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3208b565-97ac-4ed9-9d36-a289544c5fd1-metrics-tls\") pod \"dns-default-fnrlr\" (UID: \"3208b565-97ac-4ed9-9d36-a289544c5fd1\") " pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527792 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-ovn-kubernetes\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.527824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527836 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-run-ovn\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527879 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-device-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527921 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-bin\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527957 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-node-log\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527968 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nmrh6\" (UID: \"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.527991 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f69fc23e-3928-4ff3-8c03-26c5b5aca388-host-slash\") pod \"iptables-alerter-jk8c2\" (UID: \"f69fc23e-3928-4ff3-8c03-26c5b5aca388\") " pod="openshift-network-operator/iptables-alerter-jk8c2" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528034 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88b1723f-92ef-412a-a3ba-8742fa8b1131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-modprobe-d\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528099 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-multus-certs\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528129 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-run-netns\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528193 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-systemd\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528194 5024 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528237 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfe91e54-4f38-43ac-a520-2fc8e083bea9-etc-kubernetes\") pod \"tuned-rz2lx\" (UID: \"cfe91e54-4f38-43ac-a520-2fc8e083bea9\") " pod="openshift-cluster-node-tuning-operator/tuned-rz2lx" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528277 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-systemd-units\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528288 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.028267871 +0000 UTC m=+1.848979642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528322 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528358 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-hostroot\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528437 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dca34023-3797-4998-9674-ee14daf13450-dbus\") pod \"global-pull-secret-syncer-jzwlg\" (UID: \"dca34023-3797-4998-9674-ee14daf13450\") " pod="kube-system/global-pull-secret-syncer-jzwlg" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528532 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-log-socket\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528636 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/712baeaa-edb0-45ce-bcd4-9a29c7a5eec0-host-cni-netd\") pod \"ovnkube-node-lkmzz\" (UID: \"712baeaa-edb0-45ce-bcd4-9a29c7a5eec0\") " pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528670 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-cnibin\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528817 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dea29c6-0a9e-4f81-8237-f9199715657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wtdk\" (UID: \"7dea29c6-0a9e-4f81-8237-f9199715657f\") " pod="openshift-multus/multus-additional-cni-plugins-4wtdk" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528865 5024 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528933 5024 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528957 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.028940495 +0000 UTC m=+1.849652255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.528865 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-multus-conf-dir\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:50.528995 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.02897842 +0000 UTC m=+1.849690191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.529021 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8385728-e45b-41c9-b2cb-cce86024723b-host-run-k8s-cni-cncf-io\") pod \"multus-xb89g\" (UID: \"d8385728-e45b-41c9-b2cb-cce86024723b\") " pod="openshift-multus/multus-xb89g" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.529046 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.529083 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2166c14d-2366-4925-9576-1fccab9df12c-sys-fs\") pod \"aws-ebs-csi-driver-node-jmdgd\" (UID: \"2166c14d-2366-4925-9576-1fccab9df12c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmdgd" Oct 30 18:00:50.529229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.529115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78ec231-1951-451f-8ec8-b70e20b98fc5-service-ca-bundle\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.530156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.529615 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78ec231-1951-451f-8ec8-b70e20b98fc5-metrics-certs\") pod \"router-default-7765667bbf-4zpgd\" (UID: \"b78ec231-1951-451f-8ec8-b70e20b98fc5\") " pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.530217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.530188 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655bf5c4-4149-4e0e-b03b-10a41946b0f1-metrics-certs\") pod \"network-metrics-daemon-79lsr\" (UID: \"655bf5c4-4149-4e0e-b03b-10a41946b0f1\") " pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.530385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.530366 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"image-registry-6f75c69b8c-gmvng\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.551355 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.551333 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f72ft_8cfe69b0-a077-4fd0-9f7e-a513c1b882bf/dns-node-resolver/0.log" Oct 30 18:00:50.604112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.604090 5024 scope.go:122] "RemoveContainer" containerID="a9de869e920a5b59caa01237a821d950fcd4765c816a4ff37e7b2cee44502a8f" Oct 30 18:00:50.604254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.604171 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:50.604254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.604248 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" Oct 30 18:00:50.605828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.605806 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:00:50.610451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.610425 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:00:50.612208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.612186 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-79lsr" Oct 30 18:00:50.614019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.614002 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:50.956499 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.956469 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7765667bbf-4zpgd"] Oct 30 18:00:50.959362 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:50.959328 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78ec231_1951_451f_8ec8_b70e20b98fc5.slice/crio-0d293b5b567b61c7eee5e89c064152a818a20f95322edeae5f06ab6b6e3cfe50 WatchSource:0}: Error finding container 0d293b5b567b61c7eee5e89c064152a818a20f95322edeae5f06ab6b6e3cfe50: Status 404 returned error can't find the container with id 0d293b5b567b61c7eee5e89c064152a818a20f95322edeae5f06ab6b6e3cfe50 Oct 30 18:00:50.970437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.970409 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:00:50.973213 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:50.973191 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8528ac59_25e8_435f_9d93_599e19af51c5.slice/crio-81245abf688bc110533c3996679482083440d2dbda5d6a36375254fd6ea10d78 WatchSource:0}: Error finding container 81245abf688bc110533c3996679482083440d2dbda5d6a36375254fd6ea10d78: Status 404 returned error can't find the container with id 81245abf688bc110533c3996679482083440d2dbda5d6a36375254fd6ea10d78 Oct 30 18:00:50.977181 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.977145 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-79lsr"] Oct 30 18:00:50.980258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.980223 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6"] Oct 30 18:00:50.982839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.982816 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fnrlr"] Oct 30 18:00:50.992583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.992546 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:00:50.997461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:50.997337 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:00:51.035418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.035379 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:51.035532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.035457 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:51.035591 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.035548 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:51.035722 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035707 5024 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:51.035781 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035772 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:52.035753314 +0000 UTC m=+2.856465082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:51.035840 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035830 5024 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:51.035879 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035864 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:52.035854144 +0000 UTC m=+2.856565911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:51.035914 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035909 5024 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:51.035943 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:51.035938 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:52.035928525 +0000 UTC m=+2.856640292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:51.374486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.374412 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:00:51.375116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.375088 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" event={"ID":"d861baa6-45e0-44b6-9336-1ed846e00b49","Type":"ContainerStarted","Data":"6643235b56a10fabb857cebc2efc2285165119266f2c2da7e931fa34cc35cfc4"} Oct 30 18:00:51.378331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.378274 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" event={"ID":"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd","Type":"ContainerStarted","Data":"35bbc30e6dfe3359252138156a895c873782859eea37226c0b2fad0a5d89bcb0"} Oct 30 18:00:51.379577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.379509 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnrlr" event={"ID":"3208b565-97ac-4ed9-9d36-a289544c5fd1","Type":"ContainerStarted","Data":"8883372621aa6d99b06848a299974486c5bfc90d8a6f218aa2272b0517570b73"} Oct 30 18:00:51.381816 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.381511 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7765667bbf-4zpgd" event={"ID":"b78ec231-1951-451f-8ec8-b70e20b98fc5","Type":"ContainerStarted","Data":"975bf23acd167e85ca305b62f6d8ae072899ec591e59f954b132010e7ba7c67e"} Oct 30 18:00:51.381816 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.381549 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7765667bbf-4zpgd" event={"ID":"b78ec231-1951-451f-8ec8-b70e20b98fc5","Type":"ContainerStarted","Data":"0d293b5b567b61c7eee5e89c064152a818a20f95322edeae5f06ab6b6e3cfe50"} Oct 30 18:00:51.397573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.396982 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" event={"ID":"3f409e7d-56d0-43ed-ac57-5724283f1a57","Type":"ContainerStarted","Data":"8a22b942a1fbabbf8ed367eddea6243c1563cfd9b9bba4b342b1fcd61a2fc255"} Oct 30 18:00:51.397573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.397053 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" event={"ID":"3f409e7d-56d0-43ed-ac57-5724283f1a57","Type":"ContainerStarted","Data":"739a827ca38d2c6dc275c34eddab9d07798fe80638709578e45de0779afc36d0"} Oct 30 18:00:51.399726 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.399690 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" event={"ID":"8528ac59-25e8-435f-9d93-599e19af51c5","Type":"ContainerStarted","Data":"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1"} Oct 30 18:00:51.399838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.399739 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" event={"ID":"8528ac59-25e8-435f-9d93-599e19af51c5","Type":"ContainerStarted","Data":"81245abf688bc110533c3996679482083440d2dbda5d6a36375254fd6ea10d78"} Oct 30 18:00:51.401410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.401384 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79lsr" event={"ID":"655bf5c4-4149-4e0e-b03b-10a41946b0f1","Type":"ContainerStarted","Data":"8b15aeaec6414a0a4a8ba970c51db7c0ca4813cfda83810d606ffa8027e4fa7b"} Oct 30 18:00:51.605272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.605235 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:51.608083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.608057 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:51.953191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:51.953059 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzdgf_7d5af711-aeb9-4984-b639-1d6b13db2909/node-ca/0.log" Oct 30 18:00:52.051052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.051016 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:52.051243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.051114 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:52.051243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.051160 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:52.051243 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051187 5024 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:52.051404 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051255 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:54.051238575 +0000 UTC m=+4.871950346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:52.051404 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051271 5024 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:52.051404 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051312 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls podName:9fe43c59-0934-4c18-84f1-381abdcf553d nodeName:}" failed. No retries permitted until 2026-10-30 18:00:54.051300721 +0000 UTC m=+4.872012474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r5zhx" (UID: "9fe43c59-0934-4c18-84f1-381abdcf553d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:00:52.051404 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051354 5024 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:00:52.051404 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:52.051387 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert podName:16a01d63-08e8-4859-95c8-b89108043583 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:54.051377012 +0000 UTC m=+4.872088769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pllqc" (UID: "16a01d63-08e8-4859-95c8-b89108043583") : secret "networking-console-plugin-cert" not found Oct 30 18:00:52.152521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.152487 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log" Oct 30 18:00:52.352093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.352034 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dq8ks_47517669-cee7-4488-827e-b605f1cbfbc0/serve-healthcheck-canary/0.log" Oct 30 18:00:52.407849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.407829 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:52.408435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.408418 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:52.552924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.552853 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mzqd9_e0bcf238-c8de-42ac-a905-348017d59ff8/migrator/0.log" Oct 30 18:00:52.752572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.752545 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-v8jsn_cb64abf9-519f-4791-b5fc-645b72981f53/migrator/0.log" Oct 30 18:00:52.953973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:52.953942 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w_891a4e9b-107c-45f6-bef6-d825fe05e079/kube-storage-version-migrator-operator/0.log" Oct 30 18:00:53.413364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.413321 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79lsr" event={"ID":"655bf5c4-4149-4e0e-b03b-10a41946b0f1","Type":"ContainerStarted","Data":"b493e18fac5aad8a58f5248fb53e21b3d1b487d764a4a7aa3fe50c983f5a02e2"} Oct 30 18:00:53.413800 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.413396 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-79lsr" event={"ID":"655bf5c4-4149-4e0e-b03b-10a41946b0f1","Type":"ContainerStarted","Data":"380a3edeab4daa89800100b742d17232f0788bbe137e6221e0622c655de5c4a4"} Oct 30 18:00:53.415315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.415291 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" event={"ID":"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd","Type":"ContainerStarted","Data":"3437bc3155303356f0521c38d6758bce2da295f34074590556b5211c5041b31b"} Oct 30 18:00:53.415390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.415335 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nmrh6" event={"ID":"2b0ad6f7-9d36-41bc-9136-37a5a37cc0fd","Type":"ContainerStarted","Data":"357db7f9a099f4832d278b47df0749eb40845b10ab821b33ab002b17aace17bf"} Oct 30 18:00:53.417125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.417107 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:53.417239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.417102 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnrlr" event={"ID":"3208b565-97ac-4ed9-9d36-a289544c5fd1","Type":"ContainerStarted","Data":"ae1121c1f14c9b321c5b023e9f6b52d5c159daa92758031be53f4c6e37e46cff"} Oct 30 18:00:53.417239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.417224 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fnrlr" Oct 30 18:00:53.417313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.417241 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fnrlr" event={"ID":"3208b565-97ac-4ed9-9d36-a289544c5fd1","Type":"ContainerStarted","Data":"b455fefb83194847dc59e3da4952dd158ee15736b5f24dac93fcc4e071097c50"} Oct 30 18:00:53.817196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:53.817168 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:00:54.070737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.070696 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:54.070875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.070744 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:54.070875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.070785 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:54.070986 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:54.070878 5024 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:00:54.070986 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:00:54.070958 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls podName:88b1723f-92ef-412a-a3ba-8742fa8b1131 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:58.070942328 +0000 UTC m=+8.891654085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7bvmg" (UID: "88b1723f-92ef-412a-a3ba-8742fa8b1131") : secret "insights-runtime-extractor-tls" not found Oct 30 18:00:54.073374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.073352 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16a01d63-08e8-4859-95c8-b89108043583-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pllqc\" (UID: \"16a01d63-08e8-4859-95c8-b89108043583\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:54.073523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.073499 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fe43c59-0934-4c18-84f1-381abdcf553d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r5zhx\" (UID: \"9fe43c59-0934-4c18-84f1-381abdcf553d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:54.204610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.204572 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" Oct 30 18:00:54.206735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.206716 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" Oct 30 18:00:54.422730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.422692 5024 generic.go:356] "Generic (PLEG): container finished" podID="0f079577-e6f9-463f-9be1-a1d411696045" containerID="319e4e2093f1b430e7a74e76057c28e0876750221f102924a7a1b333c571dd84" exitCode=0 Oct 30 18:00:54.423167 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.422763 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerDied","Data":"319e4e2093f1b430e7a74e76057c28e0876750221f102924a7a1b333c571dd84"} Oct 30 18:00:54.423167 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.423141 5024 scope.go:122] "RemoveContainer" containerID="319e4e2093f1b430e7a74e76057c28e0876750221f102924a7a1b333c571dd84" Oct 30 18:00:54.424399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.424365 5024 generic.go:356] "Generic (PLEG): container finished" podID="891a4e9b-107c-45f6-bef6-d825fe05e079" containerID="e390538830a11d135759decd8d6a556a6d10d1d45e818317aa099d0232fb17ac" exitCode=0 Oct 30 18:00:54.424513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.424436 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" event={"ID":"891a4e9b-107c-45f6-bef6-d825fe05e079","Type":"ContainerDied","Data":"e390538830a11d135759decd8d6a556a6d10d1d45e818317aa099d0232fb17ac"} Oct 30 18:00:54.425128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.425111 5024 scope.go:122] "RemoveContainer" containerID="e390538830a11d135759decd8d6a556a6d10d1d45e818317aa099d0232fb17ac" Oct 30 18:00:54.506537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.506333 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pllqc"] Oct 30 18:00:54.542116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.542044 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx"] Oct 30 18:00:54.547479 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:54.547453 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe43c59_0934_4c18_84f1_381abdcf553d.slice/crio-f8170279ffaf5dd80db8eaf23b93fd7711d703cbe2ed0aa342b97b43fc7dca2c WatchSource:0}: Error finding container f8170279ffaf5dd80db8eaf23b93fd7711d703cbe2ed0aa342b97b43fc7dca2c: Status 404 returned error can't find the container with id f8170279ffaf5dd80db8eaf23b93fd7711d703cbe2ed0aa342b97b43fc7dca2c Oct 30 18:00:54.785272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.785242 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:54.975668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.975630 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:54.975846 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.975789 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:54.981098 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:54.981069 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ts48c" Oct 30 18:00:55.431410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:55.431375 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" event={"ID":"9fe43c59-0934-4c18-84f1-381abdcf553d","Type":"ContainerStarted","Data":"f8170279ffaf5dd80db8eaf23b93fd7711d703cbe2ed0aa342b97b43fc7dca2c"} Oct 30 18:00:55.433617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:55.433561 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerStarted","Data":"61239261e347edba817abfc5e4d23c1fcec3ade0b6898b30a3e69da5f916e193"} Oct 30 18:00:55.435917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:55.435889 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-gmb9w" event={"ID":"891a4e9b-107c-45f6-bef6-d825fe05e079","Type":"ContainerStarted","Data":"2b72bcf9b91dd9feaf8245676049e61110843479a5fa86ec0cb20b3870f94ab8"} Oct 30 18:00:55.437437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:55.437407 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" event={"ID":"16a01d63-08e8-4859-95c8-b89108043583","Type":"ContainerStarted","Data":"bf7514203fdf78d298ba4459b65fa8ea8748b94c3b901e73ea1dd6a4de07bf4b"} Oct 30 18:00:56.443227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:56.443194 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pllqc" event={"ID":"16a01d63-08e8-4859-95c8-b89108043583","Type":"ContainerStarted","Data":"96a7776e9017951c8f3b1aade76bcbada87a38cbd0d6cbee3baeb688813fdb98"} Oct 30 18:00:57.448833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:57.448793 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r5zhx" event={"ID":"9fe43c59-0934-4c18-84f1-381abdcf553d","Type":"ContainerStarted","Data":"20fc947d673a97b4f4b262059e1ea658ba28db8581be5230156b903f15857707"} Oct 30 18:00:58.107907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.107866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:58.110269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.110241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88b1723f-92ef-412a-a3ba-8742fa8b1131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bvmg\" (UID: \"88b1723f-92ef-412a-a3ba-8742fa8b1131\") " pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:58.115275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.115252 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bvmg" Oct 30 18:00:58.407776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.407746 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bvmg"] Oct 30 18:00:58.411065 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:00:58.411026 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b1723f_92ef_412a_a3ba_8742fa8b1131.slice/crio-58f23e174c8092b7c91a4b71d0cf1d5ec8cbd3a3b6fd652af9b42474c6e8072b WatchSource:0}: Error finding container 58f23e174c8092b7c91a4b71d0cf1d5ec8cbd3a3b6fd652af9b42474c6e8072b: Status 404 returned error can't find the container with id 58f23e174c8092b7c91a4b71d0cf1d5ec8cbd3a3b6fd652af9b42474c6e8072b Oct 30 18:00:58.459578 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.459541 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bvmg" event={"ID":"88b1723f-92ef-412a-a3ba-8742fa8b1131","Type":"ContainerStarted","Data":"58f23e174c8092b7c91a4b71d0cf1d5ec8cbd3a3b6fd652af9b42474c6e8072b"} Oct 30 18:00:58.594169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.594135 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:58.594324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.594269 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:58.595530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:58.595507 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7765667bbf-4zpgd" Oct 30 18:00:59.464516 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:59.464482 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bvmg" event={"ID":"88b1723f-92ef-412a-a3ba-8742fa8b1131","Type":"ContainerStarted","Data":"529b6289c1b7b54787155ef4e8d2263c766fab954b9907f60963c9c2c7684ecb"} Oct 30 18:00:59.465003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:59.464534 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bvmg" event={"ID":"88b1723f-92ef-412a-a3ba-8742fa8b1131","Type":"ContainerStarted","Data":"f92e1bd605de9226f0c1352fce6b73219bc121bd52a5248fd9070c7431d0c556"} Oct 30 18:00:59.893270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:59.893237 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:00:59.893518 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:00:59.893441 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:01:00.210192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:00.210121 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:01:00.306105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:00.306071 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:01:01.473804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:01.473765 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bvmg" event={"ID":"88b1723f-92ef-412a-a3ba-8742fa8b1131","Type":"ContainerStarted","Data":"ba107afe128f1fbd2452fcdfb998af8303dd3c401e62dab77dd17d84214f7290"} Oct 30 18:01:03.087908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:03.087870 5024 patch_prober.go:29] interesting pod/image-registry-85bf557f7d-4tfww container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 18:01:03.088291 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:03.087926 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:01:03.427534 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:03.427507 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fnrlr" Oct 30 18:01:04.180020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:04.179992 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dw7mk" Oct 30 18:01:07.312151 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:07.312121 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lkmzz" Oct 30 18:01:08.436279 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.436245 5024 patch_prober.go:29] interesting pod/image-registry-6f75c69b8c-gmvng container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 18:01:08.436773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.436297 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:01:08.846252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.846177 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-j67kw"] Oct 30 18:01:08.852071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.852040 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz"] Oct 30 18:01:08.852289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.852150 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:08.856388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.856363 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:08.856524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.856480 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:08.862067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.861748 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:01:08.862067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.861882 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:08.868335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.868304 5024 patch_prober.go:29] interesting pod/image-registry-85bf557f7d-4tfww container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 18:01:08.868487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.868369 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:01:08.870448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.870424 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:01:08.870548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.870506 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:01:08.870548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.870509 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-62n4h\"" Oct 30 18:01:08.870683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.870669 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 18:01:08.875403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.875352 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p4hk2\"" Oct 30 18:01:08.885059 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.885035 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:08.885883 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.885864 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz"] Oct 30 18:01:08.886638 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.886618 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j67kw"] Oct 30 18:01:08.889719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.889675 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 18:01:08.889834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.889763 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 18:01:08.889834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.889820 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 18:01:08.941702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.941674 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644f4f5cdc-jgpvh"] Oct 30 18:01:08.945278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.945255 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:08.965763 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.965737 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644f4f5cdc-jgpvh"] Oct 30 18:01:08.970304 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.970232 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 18:01:08.970304 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.970246 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d6sfq\"" Oct 30 18:01:08.970489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:08.970388 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 18:01:09.007252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007221 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp67\" (UniqueName: \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007257 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7fr\" (UniqueName: \"kubernetes.io/projected/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-kube-api-access-pp7fr\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.007417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007282 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-tmp\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.007521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007432 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007464 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007516 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007663 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007532 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b70074d2-074f-4c37-99e8-841f060c98b8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-59zkz\" (UID: \"b70074d2-074f-4c37-99e8-841f060c98b8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:09.007663 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007552 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007663 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.007768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.007688 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108440 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108475 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108500 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108521 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b70074d2-074f-4c37-99e8-841f060c98b8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-59zkz\" (UID: \"b70074d2-074f-4c37-99e8-841f060c98b8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108584 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-installation-pull-secrets\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108642 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-certificates\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108718 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108769 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108806 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-image-registry-private-configuration\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.108878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108864 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp67\" (UniqueName: \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.109245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108889 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7fr\" (UniqueName: \"kubernetes.io/projected/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-kube-api-access-pp7fr\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.109245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108900 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.109245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108918 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-ca-trust-extracted\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.109245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.108966 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9lm\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-kube-api-access-hj9lm\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.109466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109248 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-tls\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.109522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109463 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.109522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109486 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-tmp\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.109657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109566 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-bound-sa-token\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.109657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109646 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-trusted-ca\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.109885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109865 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-tmp\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.110018 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.109997 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.110140 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.110118 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.111461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.111442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.111547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.111485 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b70074d2-074f-4c37-99e8-841f060c98b8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-59zkz\" (UID: \"b70074d2-074f-4c37-99e8-841f060c98b8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:09.111547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.111494 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.127698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.127670 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7fr\" (UniqueName: \"kubernetes.io/projected/6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035-kube-api-access-pp7fr\") pod \"downloads-6dd789c8b7-j67kw\" (UID: \"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035\") " pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.127870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.127855 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp67\" (UniqueName: \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\") pod \"console-5568b7f495-s5sqj\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.172790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.172761 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:09.183350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.183323 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:09.190475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.190448 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210176 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-bound-sa-token\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210248 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-trusted-ca\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210306 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-installation-pull-secrets\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210523 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-certificates\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210581 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-image-registry-private-configuration\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210665 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-ca-trust-extracted\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9lm\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-kube-api-access-hj9lm\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.210730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.210701 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-tls\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.211222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.211108 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-ca-trust-extracted\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.213017 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.212967 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-certificates\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.213534 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.213489 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-trusted-ca\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.213956 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.213935 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-registry-tls\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.214311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.214286 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-image-registry-private-configuration\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.214464 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.214442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-installation-pull-secrets\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.235414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.235302 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-bound-sa-token\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.235507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.235413 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9lm\" (UniqueName: \"kubernetes.io/projected/18db95bd-a7e5-49dc-9696-2e9b3fce9e4f-kube-api-access-hj9lm\") pod \"image-registry-644f4f5cdc-jgpvh\" (UID: \"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f\") " pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.257538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.257468 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:09.489471 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.489441 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-j67kw"] Oct 30 18:01:09.492000 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:09.491961 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eba5f4f_710d_4b2f_8efc_5c2a8e5b1035.slice/crio-b9de5c51d56998d23dee9a593b099e93b327edc6b1ecc3c0a65afe9c53a22a60 WatchSource:0}: Error finding container b9de5c51d56998d23dee9a593b099e93b327edc6b1ecc3c0a65afe9c53a22a60: Status 404 returned error can't find the container with id b9de5c51d56998d23dee9a593b099e93b327edc6b1ecc3c0a65afe9c53a22a60 Oct 30 18:01:09.501727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.501707 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz"] Oct 30 18:01:09.504271 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:09.504240 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70074d2_074f_4c37_99e8_841f060c98b8.slice/crio-b329e5abc2622188678856af365fca5bf63738a21d219f37711f94539b6465a6 WatchSource:0}: Error finding container b329e5abc2622188678856af365fca5bf63738a21d219f37711f94539b6465a6: Status 404 returned error can't find the container with id b329e5abc2622188678856af365fca5bf63738a21d219f37711f94539b6465a6 Oct 30 18:01:09.506908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.506880 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j67kw" event={"ID":"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035","Type":"ContainerStarted","Data":"b9de5c51d56998d23dee9a593b099e93b327edc6b1ecc3c0a65afe9c53a22a60"} Oct 30 18:01:09.515373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.515309 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:09.517622 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:09.517572 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5cc511_d0b1_4f7d_96bc_0f0615bca3c9.slice/crio-accdda5afa2908936b4ee0161975d0fdba3940b89e0864af0560182acab31da2 WatchSource:0}: Error finding container accdda5afa2908936b4ee0161975d0fdba3940b89e0864af0560182acab31da2: Status 404 returned error can't find the container with id accdda5afa2908936b4ee0161975d0fdba3940b89e0864af0560182acab31da2 Oct 30 18:01:09.576768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:09.576718 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644f4f5cdc-jgpvh"] Oct 30 18:01:09.578898 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:09.578871 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18db95bd_a7e5_49dc_9696_2e9b3fce9e4f.slice/crio-78c063a77b58052f84a691908ccbba3c13e89f2ea9f1ffdd4721d5ab464d8fa7 WatchSource:0}: Error finding container 78c063a77b58052f84a691908ccbba3c13e89f2ea9f1ffdd4721d5ab464d8fa7: Status 404 returned error can't find the container with id 78c063a77b58052f84a691908ccbba3c13e89f2ea9f1ffdd4721d5ab464d8fa7 Oct 30 18:01:10.314091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.314052 5024 patch_prober.go:29] interesting pod/image-registry-6f75c69b8c-gmvng container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 18:01:10.314285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.314116 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:01:10.463645 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:01:10.463578 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:01:10.517636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.517455 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" event={"ID":"b70074d2-074f-4c37-99e8-841f060c98b8","Type":"ContainerStarted","Data":"b329e5abc2622188678856af365fca5bf63738a21d219f37711f94539b6465a6"} Oct 30 18:01:10.519772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.519735 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5568b7f495-s5sqj" event={"ID":"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9","Type":"ContainerStarted","Data":"accdda5afa2908936b4ee0161975d0fdba3940b89e0864af0560182acab31da2"} Oct 30 18:01:10.521876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.521846 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" event={"ID":"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f","Type":"ContainerStarted","Data":"7e5b0cd83ec50f18ad2012e37be6e9e232375f2b5b1bbf00658f43145a8200b3"} Oct 30 18:01:10.522021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.521896 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" event={"ID":"18db95bd-a7e5-49dc-9696-2e9b3fce9e4f","Type":"ContainerStarted","Data":"78c063a77b58052f84a691908ccbba3c13e89f2ea9f1ffdd4721d5ab464d8fa7"} Oct 30 18:01:10.522714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.522690 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:10.583077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:10.582789 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" podStartSLOduration=2.582750175 podStartE2EDuration="2.582750175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:10.57244533 +0000 UTC m=+21.393157106" watchObservedRunningTime="2026-10-30 18:01:10.582750175 +0000 UTC m=+21.403461952" Oct 30 18:01:11.060333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.059976 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:01:11.064040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.064013 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.097720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.097677 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 18:01:11.098785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.098759 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:01:11.128408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128302 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128376 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fmt\" (UniqueName: \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128429 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128525 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128567 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128673 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128761 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.128912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.128807 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229524 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229672 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fmt\" (UniqueName: \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229703 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229766 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229817 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229843 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229869 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.229951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.229893 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.231547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.230844 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.231547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.231283 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.231547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.231465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.231547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.231505 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.231547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.231505 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.233866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.233839 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.239122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.239073 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.248296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.248267 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fmt\" (UniqueName: \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\") pod \"console-67f7d7c4bd-r9ngr\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.378843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.378550 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:11.531190 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.531150 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" event={"ID":"b70074d2-074f-4c37-99e8-841f060c98b8","Type":"ContainerStarted","Data":"5010d07e07fecfef13259e70e2578c17ab29993a74258d767053a70782985011"} Oct 30 18:01:11.531895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.531789 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:11.537475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.537447 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" Oct 30 18:01:11.566404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.565934 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-59zkz" podStartSLOduration=2.367192423 podStartE2EDuration="3.565917575s" totalImagesPullingTime="1.198725152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:11.564767185 +0000 UTC m=+22.385478962" watchObservedRunningTime="2026-10-30 18:01:11.565917575 +0000 UTC m=+22.386629385" Oct 30 18:01:11.706221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:11.706131 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:01:11.710296 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:11.710267 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ecd768_da9f_4856_ac69_c8e30cdaacd9.slice/crio-eedaf23cf382185098d3b1a4344f15449f81ee3ffb3904f7579f20e4746a1faf WatchSource:0}: Error finding container eedaf23cf382185098d3b1a4344f15449f81ee3ffb3904f7579f20e4746a1faf: Status 404 returned error can't find the container with id eedaf23cf382185098d3b1a4344f15449f81ee3ffb3904f7579f20e4746a1faf Oct 30 18:01:12.538324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.538287 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7d7c4bd-r9ngr" event={"ID":"e1ecd768-da9f-4856-ac69-c8e30cdaacd9","Type":"ContainerStarted","Data":"eedaf23cf382185098d3b1a4344f15449f81ee3ffb3904f7579f20e4746a1faf"} Oct 30 18:01:12.799175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.798814 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm"] Oct 30 18:01:12.826192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.826149 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm"] Oct 30 18:01:12.826371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.826313 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.839416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.839227 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 18:01:12.844427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.844166 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.844427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.844215 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.844427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.844252 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.844427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.844295 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb757\" (UniqueName: \"kubernetes.io/projected/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-kube-api-access-nb757\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.849068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.849044 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fp2dd\"" Oct 30 18:01:12.849288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.849271 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 18:01:12.849908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.849889 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:01:12.945640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.945590 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb757\" (UniqueName: \"kubernetes.io/projected/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-kube-api-access-nb757\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.945802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.945709 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.945802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.945741 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.945802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.945774 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.947103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.947052 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.949124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.949095 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.949379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.949357 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:12.964966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:12.964937 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb757\" (UniqueName: \"kubernetes.io/projected/bc95cac7-936c-4a1f-b38a-5aa6a8fcf851-kube-api-access-nb757\") pod \"prometheus-operator-9cd89b74c-nm5jm\" (UID: \"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:13.189146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:13.189112 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" Oct 30 18:01:13.519012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:13.518925 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm"] Oct 30 18:01:13.524467 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:13.524434 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc95cac7_936c_4a1f_b38a_5aa6a8fcf851.slice/crio-d2b4984c44def07364dc4e7b0e4db574e80ea04474f9fb3c1d170d94a2cf2cef WatchSource:0}: Error finding container d2b4984c44def07364dc4e7b0e4db574e80ea04474f9fb3c1d170d94a2cf2cef: Status 404 returned error can't find the container with id d2b4984c44def07364dc4e7b0e4db574e80ea04474f9fb3c1d170d94a2cf2cef Oct 30 18:01:13.545862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:13.545826 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" event={"ID":"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851","Type":"ContainerStarted","Data":"d2b4984c44def07364dc4e7b0e4db574e80ea04474f9fb3c1d170d94a2cf2cef"} Oct 30 18:01:13.547863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:13.547829 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5568b7f495-s5sqj" event={"ID":"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9","Type":"ContainerStarted","Data":"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19"} Oct 30 18:01:13.586652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:13.586146 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5568b7f495-s5sqj" podStartSLOduration=1.894872581 podStartE2EDuration="5.586115808s" totalImagesPullingTime="3.691243227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:13.584569578 +0000 UTC m=+24.405281367" watchObservedRunningTime="2026-10-30 18:01:13.586115808 +0000 UTC m=+24.406827583" Oct 30 18:01:14.556829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:14.556739 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7d7c4bd-r9ngr" event={"ID":"e1ecd768-da9f-4856-ac69-c8e30cdaacd9","Type":"ContainerStarted","Data":"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a"} Oct 30 18:01:14.597699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:14.597641 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67f7d7c4bd-r9ngr" podStartSLOduration=1.158797708 podStartE2EDuration="3.597621797s" totalImagesPullingTime="2.438824089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:14.596237591 +0000 UTC m=+25.416949369" watchObservedRunningTime="2026-10-30 18:01:14.597621797 +0000 UTC m=+25.418333570" Oct 30 18:01:16.569616 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:16.569553 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" event={"ID":"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851","Type":"ContainerStarted","Data":"ac72e282ed7209147d30c806f4489b830c7b43ecceb436f7d20af17fe6f00400"} Oct 30 18:01:16.570083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:16.569636 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" event={"ID":"bc95cac7-936c-4a1f-b38a-5aa6a8fcf851","Type":"ContainerStarted","Data":"9376c3d103c36b342f8d9f1ec624163e80478a7f90c8557d8d7831aacaa60144"} Oct 30 18:01:18.479483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.479428 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-nm5jm" podStartSLOduration=4.486500334 podStartE2EDuration="6.479411611s" totalImagesPullingTime="1.992911277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:16.608029093 +0000 UTC m=+27.428740868" watchObservedRunningTime="2026-10-30 18:01:18.479411611 +0000 UTC m=+29.300123387" Oct 30 18:01:18.480607 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.480560 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lvhb2"] Oct 30 18:01:18.512891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.512864 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.560868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.560800 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:01:18.560868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.560854 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:01:18.561106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.560909 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:01:18.561106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.560994 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:01:18.599859 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.599828 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-wtmp\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.599859 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.599862 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-metrics-client-ca\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.599883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-root\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600011 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600105 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpvt\" (UniqueName: \"kubernetes.io/projected/e3d39cba-f292-4a0e-99b8-12acac3cf477-kube-api-access-8vpvt\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600325 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600142 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-textfile\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600325 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600169 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-tls\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.600325 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.600206 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-sys\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.700900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.700866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-wtmp\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.700900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.700909 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-metrics-client-ca\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.700941 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-root\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.700978 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701022 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701053 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpvt\" (UniqueName: \"kubernetes.io/projected/e3d39cba-f292-4a0e-99b8-12acac3cf477-kube-api-access-8vpvt\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-wtmp\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701086 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-textfile\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701110 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-tls\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-sys\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701219 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-sys\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701592 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-metrics-client-ca\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701702 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-textfile\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701712 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.701871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.701780 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3d39cba-f292-4a0e-99b8-12acac3cf477-root\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.704139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.704115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.704342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.704321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3d39cba-f292-4a0e-99b8-12acac3cf477-node-exporter-tls\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.721784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.721760 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpvt\" (UniqueName: \"kubernetes.io/projected/e3d39cba-f292-4a0e-99b8-12acac3cf477-kube-api-access-8vpvt\") pod \"node-exporter-lvhb2\" (UID: \"e3d39cba-f292-4a0e-99b8-12acac3cf477\") " pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.868455 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.868384 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:01:18.879411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:18.879380 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvhb2" Oct 30 18:01:18.928184 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:18.928154 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d39cba_f292_4a0e_99b8_12acac3cf477.slice/crio-8c4d8fc9ff7647c2577f5b924d3e1d76e16e8fdeeb5358e1b5801f58cdf88f25 WatchSource:0}: Error finding container 8c4d8fc9ff7647c2577f5b924d3e1d76e16e8fdeeb5358e1b5801f58cdf88f25: Status 404 returned error can't find the container with id 8c4d8fc9ff7647c2577f5b924d3e1d76e16e8fdeeb5358e1b5801f58cdf88f25 Oct 30 18:01:19.191081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:19.191036 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:19.191262 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:19.191106 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:19.196481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:19.196451 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:19.586229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:19.586097 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvhb2" event={"ID":"e3d39cba-f292-4a0e-99b8-12acac3cf477","Type":"ContainerStarted","Data":"8c4d8fc9ff7647c2577f5b924d3e1d76e16e8fdeeb5358e1b5801f58cdf88f25"} Oct 30 18:01:19.591562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:19.591362 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:20.312371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:20.312340 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:01:20.593002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:20.592932 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvhb2" event={"ID":"e3d39cba-f292-4a0e-99b8-12acac3cf477","Type":"ContainerStarted","Data":"b3678187f49004b4f5c44ee681ae120ff0817382cecad619027f0dc79aaf2548"} Oct 30 18:01:21.379503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.379455 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:21.379696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.379678 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:21.386265 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.386238 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:21.602945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.601847 5024 generic.go:356] "Generic (PLEG): container finished" podID="e3d39cba-f292-4a0e-99b8-12acac3cf477" containerID="b3678187f49004b4f5c44ee681ae120ff0817382cecad619027f0dc79aaf2548" exitCode=0 Oct 30 18:01:21.602945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.602568 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvhb2" event={"ID":"e3d39cba-f292-4a0e-99b8-12acac3cf477","Type":"ContainerDied","Data":"b3678187f49004b4f5c44ee681ae120ff0817382cecad619027f0dc79aaf2548"} Oct 30 18:01:21.608527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.608504 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:01:21.734392 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:21.734358 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:22.610588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:22.610554 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvhb2" event={"ID":"e3d39cba-f292-4a0e-99b8-12acac3cf477","Type":"ContainerStarted","Data":"91c091d5ff49476b08ab31cf6eac47e2dac7fbd476189d7a84aa90a4db486c59"} Oct 30 18:01:22.611093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:22.610655 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvhb2" event={"ID":"e3d39cba-f292-4a0e-99b8-12acac3cf477","Type":"ContainerStarted","Data":"25b4b8afdea106a41f416519238d13ab3cb921b274108e44a7f1cbf587ceb403"} Oct 30 18:01:23.821182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:23.821152 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2tn9m" Oct 30 18:01:23.859685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:23.859628 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lvhb2" podStartSLOduration=4.214986199 podStartE2EDuration="5.8595911s" totalImagesPullingTime="1.537993935s" totalInitContainerRuntime="106.610966ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:22.668943198 +0000 UTC m=+33.489654977" watchObservedRunningTime="2026-10-30 18:01:23.8595911 +0000 UTC m=+34.680302855" Oct 30 18:01:27.639789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:27.639741 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-j67kw" event={"ID":"6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035","Type":"ContainerStarted","Data":"3c6792c7463ca84e07ab54125c890013b0bcac5e8fbe6a350848efccc0f5ed32"} Oct 30 18:01:27.640284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:27.640246 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:27.653765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:27.653738 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-j67kw" Oct 30 18:01:27.696812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:27.696745 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-j67kw" podStartSLOduration=2.084153277 podStartE2EDuration="19.696724898s" totalImagesPullingTime="17.612571621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:27.69556559 +0000 UTC m=+38.516277369" watchObservedRunningTime="2026-10-30 18:01:27.696724898 +0000 UTC m=+38.517436654" Oct 30 18:01:28.281545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.281504 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:01:28.286745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.286719 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.306788 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.306756 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:01:28.410721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.410636 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.410721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.410693 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.410936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.410767 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cxr\" (UniqueName: \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.410936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.410806 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.411041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.410947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.411041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.411010 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.411122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.411104 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.411222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.411133 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511744 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511813 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511856 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cxr\" (UniqueName: \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511892 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511958 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.511996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.511990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.512275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.512020 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.512275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.512045 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.512725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.512691 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.512863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.512807 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.512863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.512807 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.513185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.513162 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.514235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.514207 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.516091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.516067 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.516227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.516171 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.529722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.529697 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cxr\" (UniqueName: \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\") pod \"console-565d4f784b-zhmnt\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.601211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.601129 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:28.911685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:28.910972 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:01:28.914328 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:01:28.914299 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadef78b_5b9f_494b_98a6_7e5039064cdb.slice/crio-b45c3a6f04c19b35238e1dad635c139df33bca7b933b04392606cef48f2d833a WatchSource:0}: Error finding container b45c3a6f04c19b35238e1dad635c139df33bca7b933b04392606cef48f2d833a: Status 404 returned error can't find the container with id b45c3a6f04c19b35238e1dad635c139df33bca7b933b04392606cef48f2d833a Oct 30 18:01:29.654346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:29.654304 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565d4f784b-zhmnt" event={"ID":"fadef78b-5b9f-494b-98a6-7e5039064cdb","Type":"ContainerStarted","Data":"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7"} Oct 30 18:01:29.654552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:29.654372 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565d4f784b-zhmnt" event={"ID":"fadef78b-5b9f-494b-98a6-7e5039064cdb","Type":"ContainerStarted","Data":"b45c3a6f04c19b35238e1dad635c139df33bca7b933b04392606cef48f2d833a"} Oct 30 18:01:29.690829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:29.690768 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-565d4f784b-zhmnt" podStartSLOduration=1.690751504 podStartE2EDuration="1.690751504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:29.690174045 +0000 UTC m=+40.510885826" watchObservedRunningTime="2026-10-30 18:01:29.690751504 +0000 UTC m=+40.511463279" Oct 30 18:01:32.543924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:32.543889 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-644f4f5cdc-jgpvh" Oct 30 18:01:32.621707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:32.621644 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:01:33.888588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:33.888521 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" containerID="cri-o://e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1" gracePeriod=30 Oct 30 18:01:34.201311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.201275 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:01:34.386527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386491 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386557 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386663 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386776 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386837 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386880 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386928 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.386976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.386958 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") pod \"8528ac59-25e8-435f-9d93-599e19af51c5\" (UID: \"8528ac59-25e8-435f-9d93-599e19af51c5\") " Oct 30 18:01:34.388192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.387857 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:34.388192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.388158 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:34.390576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.390546 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:34.391220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.391183 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:34.391320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.391219 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "kube-api-access-npfv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:34.392531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.392491 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:34.392845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.392821 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:34.397372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.397348 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted" pod "8528ac59-25e8-435f-9d93-599e19af51c5" (UID: "8528ac59-25e8-435f-9d93-599e19af51c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:01:34.488020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.487935 5024 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-registry-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.487977 5024 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-bound-sa-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.487993 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npfv7\" (UniqueName: \"kubernetes.io/projected/8528ac59-25e8-435f-9d93-599e19af51c5-kube-api-access-npfv7\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.488009 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-trusted-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.488025 5024 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8528ac59-25e8-435f-9d93-599e19af51c5-ca-trust-extracted\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.488040 5024 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-installation-pull-secrets\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.488055 5024 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8528ac59-25e8-435f-9d93-599e19af51c5-registry-certificates\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.488327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.488065 5024 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8528ac59-25e8-435f-9d93-599e19af51c5-image-registry-private-configuration\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:34.686055 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.686016 5024 generic.go:356] "Generic (PLEG): container finished" podID="8528ac59-25e8-435f-9d93-599e19af51c5" containerID="e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1" exitCode=0 Oct 30 18:01:34.686252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.686068 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" event={"ID":"8528ac59-25e8-435f-9d93-599e19af51c5","Type":"ContainerDied","Data":"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1"} Oct 30 18:01:34.686252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.686116 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" event={"ID":"8528ac59-25e8-435f-9d93-599e19af51c5","Type":"ContainerDied","Data":"81245abf688bc110533c3996679482083440d2dbda5d6a36375254fd6ea10d78"} Oct 30 18:01:34.686252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.686145 5024 scope.go:122] "RemoveContainer" containerID="e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1" Oct 30 18:01:34.686411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.686304 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85bf557f7d-4tfww" Oct 30 18:01:34.706338 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.706310 5024 scope.go:122] "RemoveContainer" containerID="e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1" Oct 30 18:01:34.709802 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:01:34.706745 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1\": container with ID starting with e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1 not found: ID does not exist" containerID="e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1" Oct 30 18:01:34.709802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.706792 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1"} err="failed to get container status \"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1\": rpc error: code = NotFound desc = could not find container \"e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1\": container with ID starting with e7279c0f104357cae847567e8d427de9c9d8d550ae01c983a70878a1e4bdaae1 not found: ID does not exist" Oct 30 18:01:34.747590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.747508 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:01:34.758034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:34.758002 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85bf557f7d-4tfww"] Oct 30 18:01:35.304710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:35.304679 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" path="/var/lib/kubelet/pods/8528ac59-25e8-435f-9d93-599e19af51c5/volumes" Oct 30 18:01:38.601521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:38.601480 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:38.602004 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:38.601552 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:38.607273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:38.607247 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:38.708732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:38.708702 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:01:38.776778 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:38.776741 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:01:40.456115 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:01:40.456078 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:01:46.761698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:46.761637 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5568b7f495-s5sqj" podUID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" containerName="console" containerID="cri-o://68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19" gracePeriod=15 Oct 30 18:01:47.070747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.070720 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:47.102681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102648 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102685 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102722 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102742 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\" (UniqueName: \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102799 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102835 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.102866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.102861 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\") pod \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\" (UID: \"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9\") " Oct 30 18:01:47.103201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.103158 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:47.103201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.103180 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:47.103201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.103188 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:47.103431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.103325 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:01:47.105140 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.105113 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "kube-api-access-bwp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:47.105242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.105214 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:47.105242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.105226 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config" pod "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" (UID: "5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:47.204173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204137 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204166 5024 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-tmp\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204178 5024 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-oauth-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204440 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204187 5024 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204440 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204196 5024 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-console-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204440 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204204 5024 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-oauth-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.204440 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.204214 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwp67\" (UniqueName: \"kubernetes.io/projected/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9-kube-api-access-bwp67\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:47.744708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.744671 5024 generic.go:356] "Generic (PLEG): container finished" podID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" containerID="68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19" exitCode=0 Oct 30 18:01:47.744905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.744729 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5568b7f495-s5sqj" Oct 30 18:01:47.744905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.744728 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5568b7f495-s5sqj" event={"ID":"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9","Type":"ContainerDied","Data":"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19"} Oct 30 18:01:47.744905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.744799 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5568b7f495-s5sqj" event={"ID":"5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9","Type":"ContainerDied","Data":"accdda5afa2908936b4ee0161975d0fdba3940b89e0864af0560182acab31da2"} Oct 30 18:01:47.744905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.744826 5024 scope.go:122] "RemoveContainer" containerID="68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19" Oct 30 18:01:47.753403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.753382 5024 scope.go:122] "RemoveContainer" containerID="68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19" Oct 30 18:01:47.753690 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:01:47.753666 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19\": container with ID starting with 68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19 not found: ID does not exist" containerID="68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19" Oct 30 18:01:47.753767 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.753704 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19"} err="failed to get container status \"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19\": rpc error: code = NotFound desc = could not find container \"68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19\": container with ID starting with 68951ce0dae78dea1ad09e796da8874c490fe975af94f30dc293916cd0d94e19 not found: ID does not exist" Oct 30 18:01:47.788667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.788627 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:47.797910 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:47.797873 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5568b7f495-s5sqj"] Oct 30 18:01:49.303220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:49.303192 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" path="/var/lib/kubelet/pods/5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9/volumes" Oct 30 18:01:55.788052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:55.788016 5024 generic.go:356] "Generic (PLEG): container finished" podID="0f079577-e6f9-463f-9be1-a1d411696045" containerID="61239261e347edba817abfc5e4d23c1fcec3ade0b6898b30a3e69da5f916e193" exitCode=0 Oct 30 18:01:55.788052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:55.788057 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerDied","Data":"61239261e347edba817abfc5e4d23c1fcec3ade0b6898b30a3e69da5f916e193"} Oct 30 18:01:55.788621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:55.788102 5024 scope.go:122] "RemoveContainer" containerID="319e4e2093f1b430e7a74e76057c28e0876750221f102924a7a1b333c571dd84" Oct 30 18:01:55.788621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:55.788390 5024 scope.go:122] "RemoveContainer" containerID="61239261e347edba817abfc5e4d23c1fcec3ade0b6898b30a3e69da5f916e193" Oct 30 18:01:55.788719 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:01:55.788634 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-kzr8f_openshift-insights(0f079577-e6f9-463f-9be1-a1d411696045)\"" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" podUID="0f079577-e6f9-463f-9be1-a1d411696045" Oct 30 18:01:57.648013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:57.647954 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" containerID="cri-o://8a22b942a1fbabbf8ed367eddea6243c1563cfd9b9bba4b342b1fcd61a2fc255" gracePeriod=30 Oct 30 18:01:57.800021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:57.799989 5024 generic.go:356] "Generic (PLEG): container finished" podID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerID="8a22b942a1fbabbf8ed367eddea6243c1563cfd9b9bba4b342b1fcd61a2fc255" exitCode=0 Oct 30 18:01:57.800170 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:57.800064 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" event={"ID":"3f409e7d-56d0-43ed-ac57-5724283f1a57","Type":"ContainerDied","Data":"8a22b942a1fbabbf8ed367eddea6243c1563cfd9b9bba4b342b1fcd61a2fc255"} Oct 30 18:01:57.929695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:57.929666 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:01:58.009036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009001 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009099 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009137 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009176 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009396 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009237 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009396 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009266 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009396 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009299 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009396 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009333 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") pod \"3f409e7d-56d0-43ed-ac57-5724283f1a57\" (UID: \"3f409e7d-56d0-43ed-ac57-5724283f1a57\") " Oct 30 18:01:58.009736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009687 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:58.010028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.009773 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:01:58.011703 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.011659 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "kube-api-access-55cjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:58.012237 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.012211 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:58.012357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.012323 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:01:58.012421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.012360 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:58.012421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.012376 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:01:58.016320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.016294 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted" pod "3f409e7d-56d0-43ed-ac57-5724283f1a57" (UID: "3f409e7d-56d0-43ed-ac57-5724283f1a57"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:01:58.110990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.110955 5024 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f409e7d-56d0-43ed-ac57-5724283f1a57-ca-trust-extracted\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.110990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.110984 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55cjl\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-kube-api-access-55cjl\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.110990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.110994 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-trusted-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.111213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.111004 5024 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-image-registry-private-configuration\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.111213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.111013 5024 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-bound-sa-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.111213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.111021 5024 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f409e7d-56d0-43ed-ac57-5724283f1a57-installation-pull-secrets\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.111213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.111029 5024 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-certificates\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.111213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.111040 5024 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f409e7d-56d0-43ed-ac57-5724283f1a57-registry-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:01:58.805239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.805201 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" event={"ID":"3f409e7d-56d0-43ed-ac57-5724283f1a57","Type":"ContainerDied","Data":"739a827ca38d2c6dc275c34eddab9d07798fe80638709578e45de0779afc36d0"} Oct 30 18:01:58.805239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.805236 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f75c69b8c-gmvng" Oct 30 18:01:58.805822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.805280 5024 scope.go:122] "RemoveContainer" containerID="8a22b942a1fbabbf8ed367eddea6243c1563cfd9b9bba4b342b1fcd61a2fc255" Oct 30 18:01:58.860216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.860187 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:01:58.870112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:58.870089 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f75c69b8c-gmvng"] Oct 30 18:01:59.304257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:01:59.304222 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" path="/var/lib/kubelet/pods/3f409e7d-56d0-43ed-ac57-5724283f1a57/volumes" Oct 30 18:02:03.798122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:03.798066 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67f7d7c4bd-r9ngr" podUID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" containerName="console" containerID="cri-o://5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a" gracePeriod=15 Oct 30 18:02:03.830854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:03.830820 5024 generic.go:356] "Generic (PLEG): container finished" podID="46e82c61-c992-432e-b1ab-dffdaadccf70" containerID="c7c2ee3e1af0166c85eb9927302a28cbe393a824f4f3e7ed8ddde8f2208e7781" exitCode=0 Oct 30 18:02:03.831013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:03.830914 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" event={"ID":"46e82c61-c992-432e-b1ab-dffdaadccf70","Type":"ContainerDied","Data":"c7c2ee3e1af0166c85eb9927302a28cbe393a824f4f3e7ed8ddde8f2208e7781"} Oct 30 18:02:03.831317 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:03.831303 5024 scope.go:122] "RemoveContainer" containerID="c7c2ee3e1af0166c85eb9927302a28cbe393a824f4f3e7ed8ddde8f2208e7781" Oct 30 18:02:04.116469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.116437 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:02:04.267822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.267791 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.267839 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.267888 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\" (UniqueName: \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.267994 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268044 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268086 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268153 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268214 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\") pod \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\" (UID: \"e1ecd768-da9f-4856-ac69-c8e30cdaacd9\") " Oct 30 18:02:04.268323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268254 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:04.268323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268265 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:04.268576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268479 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:04.268656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268625 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:04.268706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268658 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:04.268706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268688 5024 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-oauth-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.268783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268710 5024 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-tmp\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.268783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268725 5024 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.268783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.268738 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.270569 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.270547 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "kube-api-access-z2fmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:04.270879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.270860 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:04.270944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.270886 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config" pod "e1ecd768-da9f-4856-ac69-c8e30cdaacd9" (UID: "e1ecd768-da9f-4856-ac69-c8e30cdaacd9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:04.369719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.369644 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-trusted-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.369719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.369672 5024 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-oauth-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.369719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.369686 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fmt\" (UniqueName: \"kubernetes.io/projected/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-kube-api-access-z2fmt\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.369719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.369698 5024 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1ecd768-da9f-4856-ac69-c8e30cdaacd9-console-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:04.836378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.836344 5024 generic.go:356] "Generic (PLEG): container finished" podID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" containerID="5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a" exitCode=0 Oct 30 18:02:04.836865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.836398 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7d7c4bd-r9ngr" Oct 30 18:02:04.836865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.836430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7d7c4bd-r9ngr" event={"ID":"e1ecd768-da9f-4856-ac69-c8e30cdaacd9","Type":"ContainerDied","Data":"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a"} Oct 30 18:02:04.836865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.836481 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7d7c4bd-r9ngr" event={"ID":"e1ecd768-da9f-4856-ac69-c8e30cdaacd9","Type":"ContainerDied","Data":"eedaf23cf382185098d3b1a4344f15449f81ee3ffb3904f7579f20e4746a1faf"} Oct 30 18:02:04.836865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.836500 5024 scope.go:122] "RemoveContainer" containerID="5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a" Oct 30 18:02:04.839376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.839350 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xzv5l" event={"ID":"46e82c61-c992-432e-b1ab-dffdaadccf70","Type":"ContainerStarted","Data":"cd71416d7e5702440d9aab7a1c11fa3697ca9224bc072d84b6261ccff5cb2236"} Oct 30 18:02:04.845464 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.845446 5024 scope.go:122] "RemoveContainer" containerID="5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a" Oct 30 18:02:04.845763 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:02:04.845744 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a\": container with ID starting with 5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a not found: ID does not exist" containerID="5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a" Oct 30 18:02:04.845869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.845771 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a"} err="failed to get container status \"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a\": rpc error: code = NotFound desc = could not find container \"5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a\": container with ID starting with 5b213c79408ae94c41f887510ed9e805e0b774d7bc682af9bcf838952858316a not found: ID does not exist" Oct 30 18:02:04.879334 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.879305 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:02:04.889561 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:04.889530 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f7d7c4bd-r9ngr"] Oct 30 18:02:05.303296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:05.303258 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" path="/var/lib/kubelet/pods/e1ecd768-da9f-4856-ac69-c8e30cdaacd9/volumes" Oct 30 18:02:06.298235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:06.298199 5024 scope.go:122] "RemoveContainer" containerID="61239261e347edba817abfc5e4d23c1fcec3ade0b6898b30a3e69da5f916e193" Oct 30 18:02:06.850690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:06.850656 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kzr8f" event={"ID":"0f079577-e6f9-463f-9be1-a1d411696045","Type":"ContainerStarted","Data":"cd5d049a21eb1dbe180bbd8a73091ef0f32364f2ea7ffe4a79dc9a89bc01ed6c"} Oct 30 18:02:06.998254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:06.998225 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:02:06.998425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:06.998406 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" podUID="db09f543-253e-47ae-81bf-977c1e6323b7" containerName="insights-gathering" containerID="cri-o://aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28" gracePeriod=30 Oct 30 18:02:07.292662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.292623 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:02:07.399104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399075 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") pod \"db09f543-253e-47ae-81bf-977c1e6323b7\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " Oct 30 18:02:07.399646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399150 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") pod \"db09f543-253e-47ae-81bf-977c1e6323b7\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " Oct 30 18:02:07.399646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399206 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") pod \"db09f543-253e-47ae-81bf-977c1e6323b7\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " Oct 30 18:02:07.399646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399239 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") pod \"db09f543-253e-47ae-81bf-977c1e6323b7\" (UID: \"db09f543-253e-47ae-81bf-977c1e6323b7\") " Oct 30 18:02:07.399646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399631 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle" pod "db09f543-253e-47ae-81bf-977c1e6323b7" (UID: "db09f543-253e-47ae-81bf-977c1e6323b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:07.399840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399650 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path" pod "db09f543-253e-47ae-81bf-977c1e6323b7" (UID: "db09f543-253e-47ae-81bf-977c1e6323b7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:07.399840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.399669 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle" pod "db09f543-253e-47ae-81bf-977c1e6323b7" (UID: "db09f543-253e-47ae-81bf-977c1e6323b7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:07.401491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.401471 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b" pod "db09f543-253e-47ae-81bf-977c1e6323b7" (UID: "db09f543-253e-47ae-81bf-977c1e6323b7"). InnerVolumeSpecName "kube-api-access-5vd8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:07.500643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.500589 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vd8b\" (UniqueName: \"kubernetes.io/projected/db09f543-253e-47ae-81bf-977c1e6323b7-kube-api-access-5vd8b\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:07.500643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.500640 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-service-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:07.500643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.500649 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db09f543-253e-47ae-81bf-977c1e6323b7-trusted-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:07.500862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.500660 5024 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/db09f543-253e-47ae-81bf-977c1e6323b7-archives-path\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:07.855905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.855818 5024 generic.go:356] "Generic (PLEG): container finished" podID="db09f543-253e-47ae-81bf-977c1e6323b7" containerID="aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28" exitCode=2 Oct 30 18:02:07.855905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.855883 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" Oct 30 18:02:07.855905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.855890 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" event={"ID":"db09f543-253e-47ae-81bf-977c1e6323b7","Type":"ContainerDied","Data":"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28"} Oct 30 18:02:07.856132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.855931 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfb2l-ngk6j" event={"ID":"db09f543-253e-47ae-81bf-977c1e6323b7","Type":"ContainerDied","Data":"908d48a8d7412e8ad1ce496132da0ec5a51c7102e57de3c7cfe9848a33d978fc"} Oct 30 18:02:07.856132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.855950 5024 scope.go:122] "RemoveContainer" containerID="aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28" Oct 30 18:02:07.864421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.864404 5024 scope.go:122] "RemoveContainer" containerID="aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28" Oct 30 18:02:07.864698 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:02:07.864681 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28\": container with ID starting with aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28 not found: ID does not exist" containerID="aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28" Oct 30 18:02:07.864780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.864706 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28"} err="failed to get container status \"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28\": rpc error: code = NotFound desc = could not find container \"aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28\": container with ID starting with aff06cc2b6e34ba4c430cd089fab73c4dfae6af46ccd11be1b22dbc40e8efd28 not found: ID does not exist" Oct 30 18:02:07.899394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.899366 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:02:07.903167 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:07.903145 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-kfb2l-ngk6j"] Oct 30 18:02:09.303805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:09.303773 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db09f543-253e-47ae-81bf-977c1e6323b7" path="/var/lib/kubelet/pods/db09f543-253e-47ae-81bf-977c1e6323b7/volumes" Oct 30 18:02:10.454243 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:02:10.454208 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:02:15.207888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.207855 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208170 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208181 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208202 5024 cpu_manager.go:414] "containerMap: removing container" podUID="db09f543-253e-47ae-81bf-977c1e6323b7" containerName="insights-gathering" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208209 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db09f543-253e-47ae-81bf-977c1e6323b7" containerName="insights-gathering" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208217 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" containerName="console" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208222 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" containerName="console" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208240 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208245 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208252 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" containerName="console" Oct 30 18:02:15.208285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208256 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" containerName="console" Oct 30 18:02:15.208610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208308 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ecd768-da9f-4856-ac69-c8e30cdaacd9" containerName="console" Oct 30 18:02:15.208610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208326 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8528ac59-25e8-435f-9d93-599e19af51c5" containerName="registry" Oct 30 18:02:15.208610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208332 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5cc511-d0b1-4f7d-96bc-0f0615bca3c9" containerName="console" Oct 30 18:02:15.208610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208338 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f409e7d-56d0-43ed-ac57-5724283f1a57" containerName="registry" Oct 30 18:02:15.208610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.208346 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="db09f543-253e-47ae-81bf-977c1e6323b7" containerName="insights-gathering" Oct 30 18:02:15.211177 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.211159 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.242665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.242613 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:02:15.268589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268559 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268611 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268721 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268757 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2wg\" (UniqueName: \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268780 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268855 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268892 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.268965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.268925 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370233 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370249 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370276 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370294 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370322 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370340 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2wg\" (UniqueName: \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370359 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.370399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370380 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.371051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.370973 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.371160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.371111 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.371160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.371111 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.371330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.371287 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.371617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.371572 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.373031 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.373008 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.373086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.373042 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.387979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.387949 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2wg\" (UniqueName: \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\") pod \"console-746c7f77df-9g7q8\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.525618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.525517 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:15.830385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.830221 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:02:15.833794 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:02:15.833762 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb714b987_9b7d_473d_977a_567c6a207e2a.slice/crio-cdf2b5e7fedc6b665d237be557cf15993a5f36ce421c246c762c22341d0e7ec9 WatchSource:0}: Error finding container cdf2b5e7fedc6b665d237be557cf15993a5f36ce421c246c762c22341d0e7ec9: Status 404 returned error can't find the container with id cdf2b5e7fedc6b665d237be557cf15993a5f36ce421c246c762c22341d0e7ec9 Oct 30 18:02:15.893201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.893171 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746c7f77df-9g7q8" event={"ID":"b714b987-9b7d-473d-977a-567c6a207e2a","Type":"ContainerStarted","Data":"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb"} Oct 30 18:02:15.893322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:15.893216 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746c7f77df-9g7q8" event={"ID":"b714b987-9b7d-473d-977a-567c6a207e2a","Type":"ContainerStarted","Data":"cdf2b5e7fedc6b665d237be557cf15993a5f36ce421c246c762c22341d0e7ec9"} Oct 30 18:02:25.526178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:25.526144 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:25.526678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:25.526201 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:25.531127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:25.531101 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:25.568735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:25.568688 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-746c7f77df-9g7q8" podStartSLOduration=10.56867627 podStartE2EDuration="10.56867627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:15.928795032 +0000 UTC m=+86.749506810" watchObservedRunningTime="2026-10-30 18:02:25.56867627 +0000 UTC m=+96.389388122" Oct 30 18:02:25.940041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:25.940014 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:02:26.011259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:26.011227 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:02:40.453965 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:02:40.453924 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:02:42.343879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.343838 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bf7f4b65c-jmwkh"] Oct 30 18:02:42.347862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.347835 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.373005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.372978 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bf7f4b65c-jmwkh"] Oct 30 18:02:42.408358 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408316 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-oauth-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408358 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408361 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408396 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-tmp\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408457 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmp9\" (UniqueName: \"kubernetes.io/projected/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-kube-api-access-pfmp9\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408716 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-oauth-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408808 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.408992 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.408972 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-service-ca\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.409065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.409009 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-trusted-ca-bundle\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.509970 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-oauth-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510033 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-tmp\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510093 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmp9\" (UniqueName: \"kubernetes.io/projected/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-kube-api-access-pfmp9\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510136 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-oauth-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510187 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510210 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-service-ca\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510231 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-trusted-ca-bundle\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.510643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510561 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-tmp\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.511022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.510997 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.511165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.511142 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-service-ca\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.511234 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.511200 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-oauth-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.511273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.511203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-trusted-ca-bundle\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.512771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.512747 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-oauth-config\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.512987 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.512966 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-console-serving-cert\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.527606 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.527571 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmp9\" (UniqueName: \"kubernetes.io/projected/d27f19cf-63b5-497d-a0f5-27bcb1854f9d-kube-api-access-pfmp9\") pod \"console-6bf7f4b65c-jmwkh\" (UID: \"d27f19cf-63b5-497d-a0f5-27bcb1854f9d\") " pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.660583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.660544 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:42.973009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:42.972926 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bf7f4b65c-jmwkh"] Oct 30 18:02:42.976043 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:02:42.976014 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27f19cf_63b5_497d_a0f5_27bcb1854f9d.slice/crio-0da2a84c33b53923264fd0178e3e8d4adb6e19b83303e1bba2c634a88e1374bb WatchSource:0}: Error finding container 0da2a84c33b53923264fd0178e3e8d4adb6e19b83303e1bba2c634a88e1374bb: Status 404 returned error can't find the container with id 0da2a84c33b53923264fd0178e3e8d4adb6e19b83303e1bba2c634a88e1374bb Oct 30 18:02:43.008297 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:43.008265 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf7f4b65c-jmwkh" event={"ID":"d27f19cf-63b5-497d-a0f5-27bcb1854f9d","Type":"ContainerStarted","Data":"0da2a84c33b53923264fd0178e3e8d4adb6e19b83303e1bba2c634a88e1374bb"} Oct 30 18:02:44.014223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:44.014189 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf7f4b65c-jmwkh" event={"ID":"d27f19cf-63b5-497d-a0f5-27bcb1854f9d","Type":"ContainerStarted","Data":"370892f1fe920810e68ea0b4c7e5ae82d70077ad3f7534d22ae4443e8419092a"} Oct 30 18:02:44.059723 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:44.059673 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bf7f4b65c-jmwkh" podStartSLOduration=2.059658206 podStartE2EDuration="2.059658206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:44.049039394 +0000 UTC m=+114.869751171" watchObservedRunningTime="2026-10-30 18:02:44.059658206 +0000 UTC m=+114.880369999" Oct 30 18:02:51.037302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.037263 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-565d4f784b-zhmnt" podUID="fadef78b-5b9f-494b-98a6-7e5039064cdb" containerName="console" containerID="cri-o://e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7" gracePeriod=15 Oct 30 18:02:51.289750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.289685 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:02:51.389506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389465 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.389738 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389521 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\" (UniqueName: \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.389738 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389568 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.389874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389753 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.389874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389801 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.389874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389853 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.390029 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389881 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.390029 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.389908 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\") pod \"fadef78b-5b9f-494b-98a6-7e5039064cdb\" (UID: \"fadef78b-5b9f-494b-98a6-7e5039064cdb\") " Oct 30 18:02:51.390135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390031 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:51.390190 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390149 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:51.390240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390200 5024 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fadef78b-5b9f-494b-98a6-7e5039064cdb-tmp\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.390240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390215 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.390335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:51.390335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390314 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:51.390629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.390403 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:51.392021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.391992 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:51.392126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.392095 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:51.392185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.392164 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr" pod "fadef78b-5b9f-494b-98a6-7e5039064cdb" (UID: "fadef78b-5b9f-494b-98a6-7e5039064cdb"). InnerVolumeSpecName "kube-api-access-n5cxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:51.490611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490555 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cxr\" (UniqueName: \"kubernetes.io/projected/fadef78b-5b9f-494b-98a6-7e5039064cdb-kube-api-access-n5cxr\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.490611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490586 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-trusted-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.490611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490616 5024 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.490848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490626 5024 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-oauth-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.490848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490634 5024 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-oauth-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:51.490848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:51.490643 5024 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fadef78b-5b9f-494b-98a6-7e5039064cdb-console-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:02:52.048622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.048569 5024 generic.go:356] "Generic (PLEG): container finished" podID="fadef78b-5b9f-494b-98a6-7e5039064cdb" containerID="e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7" exitCode=0 Oct 30 18:02:52.049126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.048636 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565d4f784b-zhmnt" Oct 30 18:02:52.049126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.048638 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565d4f784b-zhmnt" event={"ID":"fadef78b-5b9f-494b-98a6-7e5039064cdb","Type":"ContainerDied","Data":"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7"} Oct 30 18:02:52.049126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.048752 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565d4f784b-zhmnt" event={"ID":"fadef78b-5b9f-494b-98a6-7e5039064cdb","Type":"ContainerDied","Data":"b45c3a6f04c19b35238e1dad635c139df33bca7b933b04392606cef48f2d833a"} Oct 30 18:02:52.049126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.048772 5024 scope.go:122] "RemoveContainer" containerID="e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7" Oct 30 18:02:52.057910 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.057893 5024 scope.go:122] "RemoveContainer" containerID="e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7" Oct 30 18:02:52.058192 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:02:52.058173 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7\": container with ID starting with e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7 not found: ID does not exist" containerID="e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7" Oct 30 18:02:52.058238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.058204 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7"} err="failed to get container status \"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7\": rpc error: code = NotFound desc = could not find container \"e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7\": container with ID starting with e6ec5bc7e6f9b3bb0653735b1009f9821e0ac24b53cf1846d0aa93cd4be098d7 not found: ID does not exist" Oct 30 18:02:52.102421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.102391 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:02:52.112491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.112464 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-565d4f784b-zhmnt"] Oct 30 18:02:52.661330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.661286 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:52.661507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.661363 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:52.666117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:52.666096 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:53.058445 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:53.058361 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bf7f4b65c-jmwkh" Oct 30 18:02:53.124414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:53.124374 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:02:53.303902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:02:53.303873 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadef78b-5b9f-494b-98a6-7e5039064cdb" path="/var/lib/kubelet/pods/fadef78b-5b9f-494b-98a6-7e5039064cdb/volumes" Oct 30 18:03:10.453645 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:03:10.453542 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:03:18.151132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.151060 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-746c7f77df-9g7q8" podUID="b714b987-9b7d-473d-977a-567c6a207e2a" containerName="console" containerID="cri-o://e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb" gracePeriod=15 Oct 30 18:03:18.408264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.408206 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:03:18.516486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516443 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516521 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516559 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\" (UniqueName: \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516589 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516642 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516692 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516735 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.516877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.516761 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\") pod \"b714b987-9b7d-473d-977a-567c6a207e2a\" (UID: \"b714b987-9b7d-473d-977a-567c6a207e2a\") " Oct 30 18:03:18.517207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517091 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:18.517207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517161 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:18.517207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517167 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:18.517207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517178 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:18.517207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517186 5024 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b714b987-9b7d-473d-977a-567c6a207e2a-tmp\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.517479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.517382 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:03:18.518791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.518768 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:18.518965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.518949 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:18.519104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.519086 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg" pod "b714b987-9b7d-473d-977a-567c6a207e2a" (UID: "b714b987-9b7d-473d-977a-567c6a207e2a"). InnerVolumeSpecName "kube-api-access-lf2wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:18.617879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617848 5024 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-oauth-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.617879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617876 5024 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-oauth-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.617879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617885 5024 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b714b987-9b7d-473d-977a-567c6a207e2a-console-serving-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.618112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617895 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.618112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617906 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2wg\" (UniqueName: \"kubernetes.io/projected/b714b987-9b7d-473d-977a-567c6a207e2a-kube-api-access-lf2wg\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.618112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617914 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-trusted-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:18.618112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:18.617923 5024 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b714b987-9b7d-473d-977a-567c6a207e2a-console-config\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:03:19.159478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.159443 5024 generic.go:356] "Generic (PLEG): container finished" podID="b714b987-9b7d-473d-977a-567c6a207e2a" containerID="e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb" exitCode=0 Oct 30 18:03:19.159924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.159490 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746c7f77df-9g7q8" event={"ID":"b714b987-9b7d-473d-977a-567c6a207e2a","Type":"ContainerDied","Data":"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb"} Oct 30 18:03:19.159924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.159530 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746c7f77df-9g7q8" event={"ID":"b714b987-9b7d-473d-977a-567c6a207e2a","Type":"ContainerDied","Data":"cdf2b5e7fedc6b665d237be557cf15993a5f36ce421c246c762c22341d0e7ec9"} Oct 30 18:03:19.159924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.159549 5024 scope.go:122] "RemoveContainer" containerID="e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb" Oct 30 18:03:19.159924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.159494 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746c7f77df-9g7q8" Oct 30 18:03:19.168427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.168411 5024 scope.go:122] "RemoveContainer" containerID="e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb" Oct 30 18:03:19.168756 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:03:19.168738 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb\": container with ID starting with e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb not found: ID does not exist" containerID="e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb" Oct 30 18:03:19.168819 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.168766 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb"} err="failed to get container status \"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb\": rpc error: code = NotFound desc = could not find container \"e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb\": container with ID starting with e2e80df4d56f90e112f1ed8406ac9d76c5c0c77f5d536bfe3d821fa05008badb not found: ID does not exist" Oct 30 18:03:19.202533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.202506 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:03:19.212610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.212571 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-746c7f77df-9g7q8"] Oct 30 18:03:19.303718 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:03:19.303690 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b714b987-9b7d-473d-977a-567c6a207e2a" path="/var/lib/kubelet/pods/b714b987-9b7d-473d-977a-567c6a207e2a/volumes" Oct 30 18:03:40.455248 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:03:40.455206 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:04:10.454964 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:04:10.454923 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:04:28.741931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.741736 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9"] Oct 30 18:04:28.742750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742723 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fadef78b-5b9f-494b-98a6-7e5039064cdb" containerName="console" Oct 30 18:04:28.742750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742744 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadef78b-5b9f-494b-98a6-7e5039064cdb" containerName="console" Oct 30 18:04:28.742889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742765 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b714b987-9b7d-473d-977a-567c6a207e2a" containerName="console" Oct 30 18:04:28.742889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742770 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b714b987-9b7d-473d-977a-567c6a207e2a" containerName="console" Oct 30 18:04:28.742889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742832 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadef78b-5b9f-494b-98a6-7e5039064cdb" containerName="console" Oct 30 18:04:28.742889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.742843 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b714b987-9b7d-473d-977a-567c6a207e2a" containerName="console" Oct 30 18:04:28.745933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.745917 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.768294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.768266 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:04:28.768463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.768387 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-dd5x6\"" Oct 30 18:04:28.768463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.768398 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:04:28.769617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.769578 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9"] Oct 30 18:04:28.772331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.772300 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g"] Oct 30 18:04:28.775808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.775791 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:28.792699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.792651 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g"] Oct 30 18:04:28.805070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.805040 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.805070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.805072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.805293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.805098 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5mc\" (UniqueName: \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.859734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.859706 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8s5bq\"" Oct 30 18:04:28.906238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.906430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906246 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.906430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906281 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstpg\" (UniqueName: \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:28.906430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906314 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5mc\" (UniqueName: \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.906430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906357 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:28.906641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906589 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:28.906702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906665 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.906702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.906676 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:28.924862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:28.924813 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5mc\" (UniqueName: \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:29.007958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.007875 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.007958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.007931 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.008158 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.007971 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstpg\" (UniqueName: \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.008287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.008270 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.008325 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.008299 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.025338 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.025305 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstpg\" (UniqueName: \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.062169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.062132 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:29.093732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.093698 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:29.357859 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.357831 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9"] Oct 30 18:04:29.360564 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:04:29.360529 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4547375f_f863_4538_bbbe_fddccc0e142d.slice/crio-85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b WatchSource:0}: Error finding container 85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b: Status 404 returned error can't find the container with id 85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b Oct 30 18:04:29.410948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.410908 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g"] Oct 30 18:04:29.413242 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:04:29.413213 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0654b5_2404_45d6_b96f_b65ca6bb28ed.slice/crio-ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951 WatchSource:0}: Error finding container ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951: Status 404 returned error can't find the container with id ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951 Oct 30 18:04:29.439884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.439845 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" event={"ID":"2c0654b5-2404-45d6-b96f-b65ca6bb28ed","Type":"ContainerStarted","Data":"ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951"} Oct 30 18:04:29.440962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:29.440938 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerStarted","Data":"85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b"} Oct 30 18:04:35.474533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:35.474500 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerStarted","Data":"afa7e0b3f0b585d85668ba35c82c578009f165d059a6a7b2f202160ef8170671"} Oct 30 18:04:36.480613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:36.480568 5024 generic.go:356] "Generic (PLEG): container finished" podID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerID="a2b47770656fa4e9788c9bb75dc83ad0a3da8ca05718cc844d7f338a259f7857" exitCode=0 Oct 30 18:04:36.481056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:36.480653 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" event={"ID":"2c0654b5-2404-45d6-b96f-b65ca6bb28ed","Type":"ContainerDied","Data":"a2b47770656fa4e9788c9bb75dc83ad0a3da8ca05718cc844d7f338a259f7857"} Oct 30 18:04:36.482153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:36.482133 5024 generic.go:356] "Generic (PLEG): container finished" podID="4547375f-f863-4538-bbbe-fddccc0e142d" containerID="afa7e0b3f0b585d85668ba35c82c578009f165d059a6a7b2f202160ef8170671" exitCode=0 Oct 30 18:04:36.482243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:36.482178 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerDied","Data":"afa7e0b3f0b585d85668ba35c82c578009f165d059a6a7b2f202160ef8170671"} Oct 30 18:04:37.488486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:37.488405 5024 generic.go:356] "Generic (PLEG): container finished" podID="4547375f-f863-4538-bbbe-fddccc0e142d" containerID="621d2aa086583206715b7b949f5ec3c4381ca64271193bb04b51823546ee7953" exitCode=0 Oct 30 18:04:37.488959 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:37.488491 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerDied","Data":"621d2aa086583206715b7b949f5ec3c4381ca64271193bb04b51823546ee7953"} Oct 30 18:04:37.490491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:37.490473 5024 generic.go:356] "Generic (PLEG): container finished" podID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerID="71b7e0a0c9ffa6cd6b0ef0b96533fd46fe1b48e6accec49016816f30d97074a0" exitCode=0 Oct 30 18:04:37.490567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:37.490507 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" event={"ID":"2c0654b5-2404-45d6-b96f-b65ca6bb28ed","Type":"ContainerDied","Data":"71b7e0a0c9ffa6cd6b0ef0b96533fd46fe1b48e6accec49016816f30d97074a0"} Oct 30 18:04:40.460438 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:04:40.460399 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:04:44.528229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:44.528196 5024 generic.go:356] "Generic (PLEG): container finished" podID="4547375f-f863-4538-bbbe-fddccc0e142d" containerID="f0f3192ebfb2893d88f56ea95c4b9459248797cf760601b62dc5988f2f02b3ee" exitCode=0 Oct 30 18:04:44.528718 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:44.528254 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerDied","Data":"f0f3192ebfb2893d88f56ea95c4b9459248797cf760601b62dc5988f2f02b3ee"} Oct 30 18:04:44.530379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:44.530351 5024 generic.go:356] "Generic (PLEG): container finished" podID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerID="3386dbb964c701138a3773131d9b03add8fb7360d09fe9e6c68df374123e5569" exitCode=0 Oct 30 18:04:44.530488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:44.530407 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" event={"ID":"2c0654b5-2404-45d6-b96f-b65ca6bb28ed","Type":"ContainerDied","Data":"3386dbb964c701138a3773131d9b03add8fb7360d09fe9e6c68df374123e5569"} Oct 30 18:04:45.671562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.671533 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:45.695862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.695838 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:45.780149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.780112 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\") pod \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " Oct 30 18:04:45.780337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.780232 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\" (UniqueName: \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\") pod \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " Oct 30 18:04:45.780337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.780281 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\") pod \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\" (UID: \"2c0654b5-2404-45d6-b96f-b65ca6bb28ed\") " Oct 30 18:04:45.780722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.780690 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle" pod "2c0654b5-2404-45d6-b96f-b65ca6bb28ed" (UID: "2c0654b5-2404-45d6-b96f-b65ca6bb28ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:45.782587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.782564 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg" pod "2c0654b5-2404-45d6-b96f-b65ca6bb28ed" (UID: "2c0654b5-2404-45d6-b96f-b65ca6bb28ed"). InnerVolumeSpecName "kube-api-access-bstpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:45.785618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.785538 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util" pod "2c0654b5-2404-45d6-b96f-b65ca6bb28ed" (UID: "2c0654b5-2404-45d6-b96f-b65ca6bb28ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:45.881803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.881773 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\") pod \"4547375f-f863-4538-bbbe-fddccc0e142d\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " Oct 30 18:04:45.881979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.881815 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\" (UniqueName: \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\") pod \"4547375f-f863-4538-bbbe-fddccc0e142d\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " Oct 30 18:04:45.881979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.881845 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\") pod \"4547375f-f863-4538-bbbe-fddccc0e142d\" (UID: \"4547375f-f863-4538-bbbe-fddccc0e142d\") " Oct 30 18:04:45.882051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.882021 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bstpg\" (UniqueName: \"kubernetes.io/projected/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-kube-api-access-bstpg\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.882051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.882032 5024 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-util\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.882051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.882041 5024 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c0654b5-2404-45d6-b96f-b65ca6bb28ed-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.882402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.882378 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle" pod "4547375f-f863-4538-bbbe-fddccc0e142d" (UID: "4547375f-f863-4538-bbbe-fddccc0e142d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:45.884057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.884038 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc" pod "4547375f-f863-4538-bbbe-fddccc0e142d" (UID: "4547375f-f863-4538-bbbe-fddccc0e142d"). InnerVolumeSpecName "kube-api-access-7b5mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:45.886779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.886754 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util" pod "4547375f-f863-4538-bbbe-fddccc0e142d" (UID: "4547375f-f863-4538-bbbe-fddccc0e142d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:45.983120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.983085 5024 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-util\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.983120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.983116 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5mc\" (UniqueName: \"kubernetes.io/projected/4547375f-f863-4538-bbbe-fddccc0e142d-kube-api-access-7b5mc\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:45.983120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:45.983126 5024 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4547375f-f863-4538-bbbe-fddccc0e142d-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:04:46.540383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.540356 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" Oct 30 18:04:46.540572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.540387 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjbzn9" event={"ID":"4547375f-f863-4538-bbbe-fddccc0e142d","Type":"ContainerDied","Data":"85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b"} Oct 30 18:04:46.540572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.540438 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c907d415b1e97a1423e09641524fd0561aace9be90a794a825280d401a989b" Oct 30 18:04:46.542299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.542270 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" event={"ID":"2c0654b5-2404-45d6-b96f-b65ca6bb28ed","Type":"ContainerDied","Data":"ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951"} Oct 30 18:04:46.542299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.542299 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gv56g" Oct 30 18:04:46.542486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:04:46.542318 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace2d19c1632b5e0501ae9b43e60b89dbf80ba120d8e08e5967d7072fc368951" Oct 30 18:05:10.452912 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:05:10.452868 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:05:20.599502 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599472 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j"] Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599816 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="util" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599828 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="util" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599837 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="extract" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599842 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="extract" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599850 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="util" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599854 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="util" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599871 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="pull" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599875 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="pull" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599893 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="pull" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599897 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="pull" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599903 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="extract" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599909 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="extract" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599961 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0654b5-2404-45d6-b96f-b65ca6bb28ed" containerName="extract" Oct 30 18:05:20.600019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.599969 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4547375f-f863-4538-bbbe-fddccc0e142d" containerName="extract" Oct 30 18:05:20.603310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.603285 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.638818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.638789 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j"] Oct 30 18:05:20.663517 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663489 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rnb\" (UniqueName: \"kubernetes.io/projected/cf34b5bb-0971-4897-aad1-dbb9f490d374-kube-api-access-t5rnb\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.663677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663569 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.663677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663638 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.663677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663676 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.663838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663729 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.663838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.663780 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.681541 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.681512 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:05:20.681708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.681562 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:05:20.682001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.681985 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:05:20.682001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.681993 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:05:20.682094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.681987 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:05:20.682094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.682022 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" Oct 30 18:05:20.764984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.764952 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.764995 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rnb\" (UniqueName: \"kubernetes.io/projected/cf34b5bb-0971-4897-aad1-dbb9f490d374-kube-api-access-t5rnb\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.765024 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.765046 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.765073 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.765101 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.765567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.765539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.767592 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.767563 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf34b5bb-0971-4897-aad1-dbb9f490d374-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.767756 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.767736 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.767818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.767775 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.767872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.767855 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf34b5bb-0971-4897-aad1-dbb9f490d374-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.783739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.783719 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rnb\" (UniqueName: \"kubernetes.io/projected/cf34b5bb-0971-4897-aad1-dbb9f490d374-kube-api-access-t5rnb\") pod \"opendatahub-operator-controller-manager-85c77bd65b-blr5j\" (UID: \"cf34b5bb-0971-4897-aad1-dbb9f490d374\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:20.920983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:20.920958 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:21.280834 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:05:21.280804 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf34b5bb_0971_4897_aad1_dbb9f490d374.slice/crio-b5ced8a93b72ab64d98e078acd89d60557f5fa33929b1595a3fe92117787da26 WatchSource:0}: Error finding container b5ced8a93b72ab64d98e078acd89d60557f5fa33929b1595a3fe92117787da26: Status 404 returned error can't find the container with id b5ced8a93b72ab64d98e078acd89d60557f5fa33929b1595a3fe92117787da26 Oct 30 18:05:21.284963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:21.284938 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j"] Oct 30 18:05:21.687337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:21.687301 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" event={"ID":"cf34b5bb-0971-4897-aad1-dbb9f490d374","Type":"ContainerStarted","Data":"b5ced8a93b72ab64d98e078acd89d60557f5fa33929b1595a3fe92117787da26"} Oct 30 18:05:24.702767 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:24.702733 5024 generic.go:356] "Generic (PLEG): container finished" podID="cf34b5bb-0971-4897-aad1-dbb9f490d374" containerID="c838b29f765abdfa1e9955682cfc6ec5f31b8e2629b7cd1ed79dadb3e3b06eee" exitCode=0 Oct 30 18:05:24.703215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:24.702824 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" event={"ID":"cf34b5bb-0971-4897-aad1-dbb9f490d374","Type":"ContainerDied","Data":"c838b29f765abdfa1e9955682cfc6ec5f31b8e2629b7cd1ed79dadb3e3b06eee"} Oct 30 18:05:25.710336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:25.710299 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" event={"ID":"cf34b5bb-0971-4897-aad1-dbb9f490d374","Type":"ContainerStarted","Data":"8aaba95d09be107e5b569c025d3762377169f684055cce87c1e37ef3efde8e93"} Oct 30 18:05:25.710765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:25.710396 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:25.791920 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:25.791862 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" podStartSLOduration=2.177494737 podStartE2EDuration="5.791847923s" totalImagesPullingTime="3.160016449s" totalInitContainerRuntime="454.336737ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:25.781720248 +0000 UTC m=+276.602432025" watchObservedRunningTime="2026-10-30 18:05:25.791847923 +0000 UTC m=+276.612559698" Oct 30 18:05:35.571347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.571271 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp"] Oct 30 18:05:35.576468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.576446 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.609771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.609744 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7f4dw\"" Oct 30 18:05:35.609771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.609763 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:05:35.610051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.609754 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:05:35.611203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.611177 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp"] Oct 30 18:05:35.698570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.698527 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4vp\" (UniqueName: \"kubernetes.io/projected/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-kube-api-access-bw4vp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.698570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.698570 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-tmp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.799078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.799049 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4vp\" (UniqueName: \"kubernetes.io/projected/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-kube-api-access-bw4vp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.799078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.799082 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-tmp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.799438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.799423 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-tmp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.820923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.820894 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4vp\" (UniqueName: \"kubernetes.io/projected/f09c3007-b0d1-4ee2-afb8-b86b7f6965ae-kube-api-access-bw4vp\") pod \"jobset-operator-674cd59dc5-lpzvp\" (UID: \"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:35.897779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:35.897750 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" Oct 30 18:05:36.194619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:36.194513 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp"] Oct 30 18:05:36.198539 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:05:36.198510 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf09c3007_b0d1_4ee2_afb8_b86b7f6965ae.slice/crio-5b242c87b95e54fdca03e7cf696679a259922d8202a5073c61028d8f51e3fc07 WatchSource:0}: Error finding container 5b242c87b95e54fdca03e7cf696679a259922d8202a5073c61028d8f51e3fc07: Status 404 returned error can't find the container with id 5b242c87b95e54fdca03e7cf696679a259922d8202a5073c61028d8f51e3fc07 Oct 30 18:05:36.717451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:36.717420 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-blr5j" Oct 30 18:05:36.759717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:36.759681 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" event={"ID":"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae","Type":"ContainerStarted","Data":"5b242c87b95e54fdca03e7cf696679a259922d8202a5073c61028d8f51e3fc07"} Oct 30 18:05:39.778909 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:39.778868 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" event={"ID":"f09c3007-b0d1-4ee2-afb8-b86b7f6965ae","Type":"ContainerStarted","Data":"34290ec1a969fc21d90c2f906978b5a321a04dcb25037fd9cad0b3ace5ea0b22"} Oct 30 18:05:39.819711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:39.819650 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lpzvp" podStartSLOduration=2.088360299 podStartE2EDuration="4.819635803s" totalImagesPullingTime="2.731275504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:39.81932873 +0000 UTC m=+290.640040505" watchObservedRunningTime="2026-10-30 18:05:39.819635803 +0000 UTC m=+290.640347576" Oct 30 18:05:40.454333 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:05:40.454293 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:05:47.884274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.884239 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms"] Oct 30 18:05:47.888410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.888390 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:47.899428 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.899401 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:47.899546 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.899446 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:47.899546 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.899479 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92nl\" (UniqueName: \"kubernetes.io/projected/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-kube-api-access-p92nl\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:47.904991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.904969 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:05:47.904991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.904977 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:05:47.905160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.905014 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:05:47.915489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.915455 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:05:47.915489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.915456 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" Oct 30 18:05:47.917170 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.917137 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms"] Oct 30 18:05:48.000023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:47.999979 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.000023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.000027 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.000311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.000064 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92nl\" (UniqueName: \"kubernetes.io/projected/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-kube-api-access-p92nl\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.000781 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.000754 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.002812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.002790 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.023290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.023263 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92nl\" (UniqueName: \"kubernetes.io/projected/28da65aa-8ccc-46c5-a8dc-06c341aa4e1c-kube-api-access-p92nl\") pod \"kuadrant-console-plugin-66c4fb575d-h8nms\" (UID: \"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.199703 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.199666 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" Oct 30 18:05:48.496704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.496591 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms"] Oct 30 18:05:48.501208 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:05:48.501176 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28da65aa_8ccc_46c5_a8dc_06c341aa4e1c.slice/crio-50a9f46aaca6d4802701ca91382f0b4c89ca80c89d8febd733ee3b9d20fc996d WatchSource:0}: Error finding container 50a9f46aaca6d4802701ca91382f0b4c89ca80c89d8febd733ee3b9d20fc996d: Status 404 returned error can't find the container with id 50a9f46aaca6d4802701ca91382f0b4c89ca80c89d8febd733ee3b9d20fc996d Oct 30 18:05:48.824637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:48.824533 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" event={"ID":"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c","Type":"ContainerStarted","Data":"50a9f46aaca6d4802701ca91382f0b4c89ca80c89d8febd733ee3b9d20fc996d"} Oct 30 18:05:49.320808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:49.320781 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:05:49.321276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:49.321010 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:05:52.321022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.320985 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4"] Oct 30 18:05:52.343676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.343643 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.346208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.346176 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4"] Oct 30 18:05:52.357204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.357180 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:05:52.366546 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.366518 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:05:52.366721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.366644 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8ljsd\"" Oct 30 18:05:52.437971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.437932 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.438164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.438110 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrqf\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-kube-api-access-sbrqf\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.539559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.539504 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrqf\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-kube-api-access-sbrqf\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.539779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.539588 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.557518 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.557488 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrqf\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-kube-api-access-sbrqf\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.567063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.567030 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92ce5a81-1b4a-4941-90a5-5a1e38ed60b0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tbfd4\" (UID: \"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.662924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.662878 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:52.975711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.975632 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4"] Oct 30 18:05:52.979941 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:05:52.979898 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ce5a81_1b4a_4941_90a5_5a1e38ed60b0.slice/crio-611e02a1b151e7024b6213c3991ff3493a0c51edf9759f0b53c28ccbdbd21892 WatchSource:0}: Error finding container 611e02a1b151e7024b6213c3991ff3493a0c51edf9759f0b53c28ccbdbd21892: Status 404 returned error can't find the container with id 611e02a1b151e7024b6213c3991ff3493a0c51edf9759f0b53c28ccbdbd21892 Oct 30 18:05:52.982715 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:52.982696 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:05:53.854064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:53.854017 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" event={"ID":"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0","Type":"ContainerStarted","Data":"611e02a1b151e7024b6213c3991ff3493a0c51edf9759f0b53c28ccbdbd21892"} Oct 30 18:05:54.362647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.362198 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb"] Oct 30 18:05:54.367882 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.367848 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.397445 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.397412 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb"] Oct 30 18:05:54.428629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.427654 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:05:54.429145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.429102 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:05:54.429145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.429117 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nqjgb\"" Oct 30 18:05:54.558949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.558903 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz55k\" (UniqueName: \"kubernetes.io/projected/3552bb13-3271-4be5-8d76-5bac03ce89df-kube-api-access-jz55k\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.559131 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.558981 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.559131 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.559038 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552bb13-3271-4be5-8d76-5bac03ce89df-tmp\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.660482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.660444 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz55k\" (UniqueName: \"kubernetes.io/projected/3552bb13-3271-4be5-8d76-5bac03ce89df-kube-api-access-jz55k\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.660711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.660509 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.660711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.660555 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552bb13-3271-4be5-8d76-5bac03ce89df-tmp\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.660711 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:05:54.660695 5024 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 18:05:54.660895 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:05:54.660765 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs podName:3552bb13-3271-4be5-8d76-5bac03ce89df nodeName:}" failed. No retries permitted until 2026-10-30 18:05:55.160743064 +0000 UTC m=+305.981454834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs") pod "kube-auth-proxy-69dd9bcffb-zsxmb" (UID: "3552bb13-3271-4be5-8d76-5bac03ce89df") : secret "kube-auth-proxy-tls" not found Oct 30 18:05:54.663470 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.663441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3552bb13-3271-4be5-8d76-5bac03ce89df-tmp\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.707737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.707697 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz55k\" (UniqueName: \"kubernetes.io/projected/3552bb13-3271-4be5-8d76-5bac03ce89df-kube-api-access-jz55k\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:54.862719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:54.862672 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" event={"ID":"28da65aa-8ccc-46c5-a8dc-06c341aa4e1c","Type":"ContainerStarted","Data":"4fca5a6d68e930f6a5a946dcf1d29eb5d61122901d65b345a6da39ef052fabf0"} Oct 30 18:05:55.168104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.168004 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:55.171066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.171027 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552bb13-3271-4be5-8d76-5bac03ce89df-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-zsxmb\" (UID: \"3552bb13-3271-4be5-8d76-5bac03ce89df\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:55.285762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.285723 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" Oct 30 18:05:55.614881 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.614729 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-h8nms" podStartSLOduration=3.266583221 podStartE2EDuration="8.614711674s" totalImagesPullingTime="5.348128453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:54.907241246 +0000 UTC m=+305.727953022" watchObservedRunningTime="2026-10-30 18:05:55.614711674 +0000 UTC m=+306.435423463" Oct 30 18:05:55.616202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.616174 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb"] Oct 30 18:05:55.620567 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:05:55.620537 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3552bb13_3271_4be5_8d76_5bac03ce89df.slice/crio-3e5f2454c1e865b8221879750ccbdf873840ae05cbffe96ef655716fcb738724 WatchSource:0}: Error finding container 3e5f2454c1e865b8221879750ccbdf873840ae05cbffe96ef655716fcb738724: Status 404 returned error can't find the container with id 3e5f2454c1e865b8221879750ccbdf873840ae05cbffe96ef655716fcb738724 Oct 30 18:05:55.870787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:55.870695 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" event={"ID":"3552bb13-3271-4be5-8d76-5bac03ce89df","Type":"ContainerStarted","Data":"3e5f2454c1e865b8221879750ccbdf873840ae05cbffe96ef655716fcb738724"} Oct 30 18:05:56.879506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:56.879415 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" event={"ID":"92ce5a81-1b4a-4941-90a5-5a1e38ed60b0","Type":"ContainerStarted","Data":"2bc48b039520624036eb4937622712277433806a93d98c0438486d2b25b60c6a"} Oct 30 18:05:56.879506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:56.879498 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:05:58.893192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:58.893158 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" event={"ID":"3552bb13-3271-4be5-8d76-5bac03ce89df","Type":"ContainerStarted","Data":"d1d2f10468e159afd55d689f54d9ca93c060b2d6132f6c52717917fae1767bc8"} Oct 30 18:05:58.944307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:58.944247 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" podStartSLOduration=3.419278178 podStartE2EDuration="6.944230641s" totalImagesPullingTime="3.524952463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:56.918707721 +0000 UTC m=+307.739419497" watchObservedRunningTime="2026-10-30 18:05:58.944230641 +0000 UTC m=+309.764942416" Oct 30 18:05:58.944491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:05:58.944336 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-zsxmb" podStartSLOduration=2.321186738 podStartE2EDuration="4.944331434s" totalImagesPullingTime="2.623144696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:58.941203953 +0000 UTC m=+309.761915729" watchObservedRunningTime="2026-10-30 18:05:58.944331434 +0000 UTC m=+309.765043209" Oct 30 18:06:02.887675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:02.887644 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" Oct 30 18:06:03.127724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.127670 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9"] Oct 30 18:06:03.129916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.129899 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.136942 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.136912 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.137060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.136973 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/882bc53c-8e28-44f5-8309-da5cd21ff4b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.137146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.137121 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvq5v\" (UniqueName: \"kubernetes.io/projected/882bc53c-8e28-44f5-8309-da5cd21ff4b5-kube-api-access-hvq5v\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.137247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.137224 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.146188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.146161 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:06:03.146341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.146208 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:06:03.167153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.167121 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 18:06:03.168927 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.168894 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9"] Oct 30 18:06:03.184434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.184405 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-zw5t8\"" Oct 30 18:06:03.237997 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.237962 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/882bc53c-8e28-44f5-8309-da5cd21ff4b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238020 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvq5v\" (UniqueName: \"kubernetes.io/projected/882bc53c-8e28-44f5-8309-da5cd21ff4b5-kube-api-access-hvq5v\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238074 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238507 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238547 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/882bc53c-8e28-44f5-8309-da5cd21ff4b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.238968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.238952 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/882bc53c-8e28-44f5-8309-da5cd21ff4b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.256566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.256535 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvq5v\" (UniqueName: \"kubernetes.io/projected/882bc53c-8e28-44f5-8309-da5cd21ff4b5-kube-api-access-hvq5v\") pod \"kserve-module-controller-manager-7566cf5fc8-7prn9\" (UID: \"882bc53c-8e28-44f5-8309-da5cd21ff4b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.442354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.442245 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:03.762968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.762867 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9"] Oct 30 18:06:03.768682 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:06:03.768650 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882bc53c_8e28_44f5_8309_da5cd21ff4b5.slice/crio-00623b3c0b7500a08d099aab34d668196fa4ac3b26d8397a56b5327a5996c19b WatchSource:0}: Error finding container 00623b3c0b7500a08d099aab34d668196fa4ac3b26d8397a56b5327a5996c19b: Status 404 returned error can't find the container with id 00623b3c0b7500a08d099aab34d668196fa4ac3b26d8397a56b5327a5996c19b Oct 30 18:06:03.917939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:03.917902 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" event={"ID":"882bc53c-8e28-44f5-8309-da5cd21ff4b5","Type":"ContainerStarted","Data":"00623b3c0b7500a08d099aab34d668196fa4ac3b26d8397a56b5327a5996c19b"} Oct 30 18:06:05.929675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:05.929639 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" event={"ID":"882bc53c-8e28-44f5-8309-da5cd21ff4b5","Type":"ContainerStarted","Data":"61a224dfce442d96507ba11282cfda24b7687a92528d4b0916ce11f08feb8504"} Oct 30 18:06:05.930084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:05.929772 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:05.966895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:05.966841 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" podStartSLOduration=1.28472878 podStartE2EDuration="2.96682255s" totalImagesPullingTime="1.68209377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:05.966465561 +0000 UTC m=+316.787177337" watchObservedRunningTime="2026-10-30 18:06:05.96682255 +0000 UTC m=+316.787534326" Oct 30 18:06:10.453911 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:06:10.453868 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:06:26.783154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.783118 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:26.948097 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.948068 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:26.948294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.948179 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-7prn9" Oct 30 18:06:26.948294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.948207 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:26.948294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.948278 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:26.960516 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:26.960490 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:06:27.057997 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.057912 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.058154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.058131 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxvt\" (UniqueName: \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.159434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.159393 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxvt\" (UniqueName: \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.159633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.159492 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.160080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.160062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.176935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.176906 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxvt\" (UniqueName: \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\") pod \"limitador-limitador-74dfcfb894-772hn\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.271542 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.271505 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:27.591682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:27.591646 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:27.596199 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:06:27.596115 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123dc046_5c31_4d2b_b95a_12338be93912.slice/crio-fc7a39a09ff2d3e173d3e1912b2b7f9dd4ccc9896a9ec1ff569350596edfde0c WatchSource:0}: Error finding container fc7a39a09ff2d3e173d3e1912b2b7f9dd4ccc9896a9ec1ff569350596edfde0c: Status 404 returned error can't find the container with id fc7a39a09ff2d3e173d3e1912b2b7f9dd4ccc9896a9ec1ff569350596edfde0c Oct 30 18:06:28.037821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:28.037787 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" event={"ID":"123dc046-5c31-4d2b-b95a-12338be93912","Type":"ContainerStarted","Data":"fc7a39a09ff2d3e173d3e1912b2b7f9dd4ccc9896a9ec1ff569350596edfde0c"} Oct 30 18:06:31.054420 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:31.054381 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" event={"ID":"123dc046-5c31-4d2b-b95a-12338be93912","Type":"ContainerStarted","Data":"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22"} Oct 30 18:06:31.054894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:31.054513 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:31.089092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:31.089039 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" podStartSLOduration=2.618545932 podStartE2EDuration="5.089023251s" totalImagesPullingTime="2.470477319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:31.088365423 +0000 UTC m=+341.909077199" watchObservedRunningTime="2026-10-30 18:06:31.089023251 +0000 UTC m=+341.909735027" Oct 30 18:06:40.456094 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:06:40.456053 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:06:41.028963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.028926 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:41.029203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.029154 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" podUID="123dc046-5c31-4d2b-b95a-12338be93912" containerName="limitador" containerID="cri-o://b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22" gracePeriod=30 Oct 30 18:06:41.029908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.029891 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:41.587975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.587912 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:41.596477 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.596457 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\" (UniqueName: \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\") pod \"123dc046-5c31-4d2b-b95a-12338be93912\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " Oct 30 18:06:41.596585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.596502 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\" (UniqueName: \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\") pod \"123dc046-5c31-4d2b-b95a-12338be93912\" (UID: \"123dc046-5c31-4d2b-b95a-12338be93912\") " Oct 30 18:06:41.596901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.596880 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file" pod "123dc046-5c31-4d2b-b95a-12338be93912" (UID: "123dc046-5c31-4d2b-b95a-12338be93912"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:41.598815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.598795 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt" pod "123dc046-5c31-4d2b-b95a-12338be93912" (UID: "123dc046-5c31-4d2b-b95a-12338be93912"). InnerVolumeSpecName "kube-api-access-lcxvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:41.697853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.697816 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxvt\" (UniqueName: \"kubernetes.io/projected/123dc046-5c31-4d2b-b95a-12338be93912-kube-api-access-lcxvt\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:06:41.697853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:41.697848 5024 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/123dc046-5c31-4d2b-b95a-12338be93912-config-file\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:06:42.107461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.107424 5024 generic.go:356] "Generic (PLEG): container finished" podID="123dc046-5c31-4d2b-b95a-12338be93912" containerID="b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22" exitCode=0 Oct 30 18:06:42.107461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.107464 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" Oct 30 18:06:42.107679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.107511 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" event={"ID":"123dc046-5c31-4d2b-b95a-12338be93912","Type":"ContainerDied","Data":"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22"} Oct 30 18:06:42.107679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.107564 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-772hn" event={"ID":"123dc046-5c31-4d2b-b95a-12338be93912","Type":"ContainerDied","Data":"fc7a39a09ff2d3e173d3e1912b2b7f9dd4ccc9896a9ec1ff569350596edfde0c"} Oct 30 18:06:42.107679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.107586 5024 scope.go:122] "RemoveContainer" containerID="b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22" Oct 30 18:06:42.116507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.116491 5024 scope.go:122] "RemoveContainer" containerID="b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22" Oct 30 18:06:42.116842 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:06:42.116822 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22\": container with ID starting with b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22 not found: ID does not exist" containerID="b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22" Oct 30 18:06:42.116916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.116851 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22"} err="failed to get container status \"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22\": rpc error: code = NotFound desc = could not find container \"b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22\": container with ID starting with b922ca82ce898f7776a42a67c2413dd60f3c5285d4eceaf8f4f232de2db91a22 not found: ID does not exist" Oct 30 18:06:42.154686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.154649 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:42.164359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:42.164318 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-772hn"] Oct 30 18:06:43.303950 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:43.303916 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123dc046-5c31-4d2b-b95a-12338be93912" path="/var/lib/kubelet/pods/123dc046-5c31-4d2b-b95a-12338be93912/volumes" Oct 30 18:06:58.785061 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.785025 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-db7c8"] Oct 30 18:06:58.785590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.785504 5024 cpu_manager.go:414] "containerMap: removing container" podUID="123dc046-5c31-4d2b-b95a-12338be93912" containerName="limitador" Oct 30 18:06:58.785590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.785523 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123dc046-5c31-4d2b-b95a-12338be93912" containerName="limitador" Oct 30 18:06:58.785732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.785657 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="123dc046-5c31-4d2b-b95a-12338be93912" containerName="limitador" Oct 30 18:06:58.801478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.801445 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.804436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.804406 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:06:58.807881 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.807860 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.811163 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.810737 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-db7c8"] Oct 30 18:06:58.816589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.816565 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 18:06:58.819524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.819502 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fh9d8\"" Oct 30 18:06:58.826566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.826536 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzpfj\"" Oct 30 18:06:58.829587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.829559 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:06:58.836900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.836869 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.837065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.836921 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvghn\" (UniqueName: \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.837065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.837001 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cef5aee3-add8-449c-9635-58897b7023a9-cert\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.837065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.837030 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjbs\" (UniqueName: \"kubernetes.io/projected/cef5aee3-add8-449c-9635-58897b7023a9-kube-api-access-jqjbs\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.840907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.840880 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:06:58.938239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.938199 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.938457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.938246 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvghn\" (UniqueName: \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.938457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.938297 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cef5aee3-add8-449c-9635-58897b7023a9-cert\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.938457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.938323 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjbs\" (UniqueName: \"kubernetes.io/projected/cef5aee3-add8-449c-9635-58897b7023a9-kube-api-access-jqjbs\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.941106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.941073 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cef5aee3-add8-449c-9635-58897b7023a9-cert\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.941106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.941092 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:58.956508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.956472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjbs\" (UniqueName: \"kubernetes.io/projected/cef5aee3-add8-449c-9635-58897b7023a9-kube-api-access-jqjbs\") pod \"kserve-controller-manager-776cc66d47-db7c8\" (UID: \"cef5aee3-add8-449c-9635-58897b7023a9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:58.956679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:58.956533 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvghn\" (UniqueName: \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\") pod \"llmisvc-controller-manager-869d5dcc5d-g46qt\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:59.117077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:59.116987 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:06:59.127538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:59.127285 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:06:59.424914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:59.424879 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-db7c8"] Oct 30 18:06:59.429248 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:06:59.429222 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef5aee3_add8_449c_9635_58897b7023a9.slice/crio-dae8db29672caf70e717101442ef7be0ba5b1da36f1bf6f77449faf85e1daba5 WatchSource:0}: Error finding container dae8db29672caf70e717101442ef7be0ba5b1da36f1bf6f77449faf85e1daba5: Status 404 returned error can't find the container with id dae8db29672caf70e717101442ef7be0ba5b1da36f1bf6f77449faf85e1daba5 Oct 30 18:06:59.440772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:06:59.440740 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:06:59.444874 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:06:59.444849 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4dbe9c2a_9caf_483f_ac4a_0b20a3304a98.slice/crio-580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6 WatchSource:0}: Error finding container 580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6: Status 404 returned error can't find the container with id 580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6 Oct 30 18:07:00.212753 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:00.212712 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" event={"ID":"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98","Type":"ContainerStarted","Data":"580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6"} Oct 30 18:07:00.214684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:00.214640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" event={"ID":"cef5aee3-add8-449c-9635-58897b7023a9","Type":"ContainerStarted","Data":"dae8db29672caf70e717101442ef7be0ba5b1da36f1bf6f77449faf85e1daba5"} Oct 30 18:07:02.227136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.227090 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" event={"ID":"cef5aee3-add8-449c-9635-58897b7023a9","Type":"ContainerStarted","Data":"3d3a76cd7c63f2a7c410e798d253ba3537059b02f3dad349bfdd7f26d70b787c"} Oct 30 18:07:02.227653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.227166 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:07:02.229002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.228978 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" event={"ID":"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98","Type":"ContainerStarted","Data":"4038a90f9932adcd5a85415626c00303a32ba010cf43bd41ba899211278a2df6"} Oct 30 18:07:02.229102 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.229084 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:07:02.267679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.267589 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" podStartSLOduration=1.872164582 podStartE2EDuration="4.267570048s" totalImagesPullingTime="2.395405466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:02.265460303 +0000 UTC m=+373.086172079" watchObservedRunningTime="2026-10-30 18:07:02.267570048 +0000 UTC m=+373.088281824" Oct 30 18:07:02.301654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:02.301582 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" podStartSLOduration=2.105443189 podStartE2EDuration="4.301567187s" totalImagesPullingTime="2.196123998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:02.300257907 +0000 UTC m=+373.120969685" watchObservedRunningTime="2026-10-30 18:07:02.301567187 +0000 UTC m=+373.122279012" Oct 30 18:07:10.456841 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:07:10.456800 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:07:12.819283 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.819245 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s"] Oct 30 18:07:12.824004 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.823978 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.843870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.843836 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s"] Oct 30 18:07:12.851074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.851009 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ff94j\"" Oct 30 18:07:12.851625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.851567 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 18:07:12.851770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.851670 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 18:07:12.852388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.852369 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 18:07:12.872439 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.872391 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.872749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.872726 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.873915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.873880 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.874077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.873955 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbp5\" (UniqueName: \"kubernetes.io/projected/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-kube-api-access-lhbp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.975183 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.975140 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.975360 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.975202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.975360 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.975241 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.975360 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.975285 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbp5\" (UniqueName: \"kubernetes.io/projected/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-kube-api-access-lhbp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.975360 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:07:12.975297 5024 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 18:07:12.975516 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:07:12.975364 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert podName:4536ebbb-f8ee-4a78-a0ee-d046bb448e16 nodeName:}" failed. No retries permitted until 2026-10-30 18:07:13.475345418 +0000 UTC m=+384.296057184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" (UID: "4536ebbb-f8ee-4a78-a0ee-d046bb448e16") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 18:07:12.976121 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.976085 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.976313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.976295 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:12.995395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:12.995358 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbp5\" (UniqueName: \"kubernetes.io/projected/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-kube-api-access-lhbp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:13.481207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:13.481167 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:13.483987 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:13.483961 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4536ebbb-f8ee-4a78-a0ee-d046bb448e16-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vgw6s\" (UID: \"4536ebbb-f8ee-4a78-a0ee-d046bb448e16\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:13.739353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:13.739263 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:14.036852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:14.036751 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s"] Oct 30 18:07:14.041591 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:07:14.041559 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4536ebbb_f8ee_4a78_a0ee_d046bb448e16.slice/crio-63bc656b9e6db2447fcc3b146ecb4e89d6262098d6e1381cb19d72c9d368c360 WatchSource:0}: Error finding container 63bc656b9e6db2447fcc3b146ecb4e89d6262098d6e1381cb19d72c9d368c360: Status 404 returned error can't find the container with id 63bc656b9e6db2447fcc3b146ecb4e89d6262098d6e1381cb19d72c9d368c360 Oct 30 18:07:14.288648 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:14.288532 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" event={"ID":"4536ebbb-f8ee-4a78-a0ee-d046bb448e16","Type":"ContainerStarted","Data":"63bc656b9e6db2447fcc3b146ecb4e89d6262098d6e1381cb19d72c9d368c360"} Oct 30 18:07:25.352939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:25.352888 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" event={"ID":"4536ebbb-f8ee-4a78-a0ee-d046bb448e16","Type":"ContainerStarted","Data":"7c04590ac0b6212840e75341d019da83d7edd0d1a6f0da3e5b89874736db3c42"} Oct 30 18:07:25.353412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:25.353098 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:25.388503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:25.388439 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" podStartSLOduration=2.609087324 podStartE2EDuration="13.388420391s" totalImagesPullingTime="10.779333067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:25.386719336 +0000 UTC m=+396.207431112" watchObservedRunningTime="2026-10-30 18:07:25.388420391 +0000 UTC m=+396.209132199" Oct 30 18:07:33.235943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:33.235864 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:07:33.238947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:33.238919 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-db7c8" Oct 30 18:07:36.363913 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:36.363880 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vgw6s" Oct 30 18:07:40.454569 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:07:40.454528 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:07:56.771756 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:56.771717 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:07:57.091796 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.091710 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:07:57.091796 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.091720 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.104307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.104284 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:07:57.114176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.114143 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:07:57.114315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.114195 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:07:57.114315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.114257 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:07:57.114388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.114341 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:07:57.191251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.191217 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.191251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.191258 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.191469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.191392 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pp5\" (UniqueName: \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.191469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.191425 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.292040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.291990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pp5\" (UniqueName: \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.292259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.292050 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.292259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.292083 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.292259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.292120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.292932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.292879 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.294950 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.294927 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.295044 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.294927 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.309252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.309228 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pp5\" (UniqueName: \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\") pod \"maas-api-78b5489d4d-dl8h5\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.436955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.436921 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:07:57.734105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:57.734007 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:07:57.738611 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:07:57.738561 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aa2a0b_b5f3_4088_b564_bf1a2cd687c6.slice/crio-6ccf101e25dad006732632e00ee01da0f68ca9e0b7cda6c4c013213c929c0532 WatchSource:0}: Error finding container 6ccf101e25dad006732632e00ee01da0f68ca9e0b7cda6c4c013213c929c0532: Status 404 returned error can't find the container with id 6ccf101e25dad006732632e00ee01da0f68ca9e0b7cda6c4c013213c929c0532 Oct 30 18:07:58.089707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.089624 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q"] Oct 30 18:07:58.092823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.092803 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.107971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.107945 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" Oct 30 18:07:58.117928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.117900 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:07:58.120501 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.120473 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q"] Oct 30 18:07:58.200906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.200865 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.200906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.200906 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.200926 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201049 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29abc014-6eb3-474b-95b4-cc26370fd325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201081 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201167 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201194 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7shr\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-kube-api-access-x7shr\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201375 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201270 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29abc014-6eb3-474b-95b4-cc26370fd325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.201375 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.201329 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.302809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302763 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29abc014-6eb3-474b-95b4-cc26370fd325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.302809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302814 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302858 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302892 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7shr\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-kube-api-access-x7shr\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302937 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29abc014-6eb3-474b-95b4-cc26370fd325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.302991 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.303055 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.303084 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.303110 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.303877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.303848 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306027 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306354 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29abc014-6eb3-474b-95b4-cc26370fd325-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306478 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306560 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29abc014-6eb3-474b-95b4-cc26370fd325-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.306687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.306673 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29abc014-6eb3-474b-95b4-cc26370fd325-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.321618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.321561 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7shr\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-kube-api-access-x7shr\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.324436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.324406 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29abc014-6eb3-474b-95b4-cc26370fd325-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-hn66q\" (UID: \"29abc014-6eb3-474b-95b4-cc26370fd325\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.406472 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.406434 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:07:58.525878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.525839 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" event={"ID":"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6","Type":"ContainerStarted","Data":"6ccf101e25dad006732632e00ee01da0f68ca9e0b7cda6c4c013213c929c0532"} Oct 30 18:07:58.747512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:58.746867 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q"] Oct 30 18:07:58.751106 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:07:58.751069 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29abc014_6eb3_474b_95b4_cc26370fd325.slice/crio-47e255e1629efda2b3082ebddc8e9d408ceb850e51fe34bcc5ede62fd6f2fcba WatchSource:0}: Error finding container 47e255e1629efda2b3082ebddc8e9d408ceb850e51fe34bcc5ede62fd6f2fcba: Status 404 returned error can't find the container with id 47e255e1629efda2b3082ebddc8e9d408ceb850e51fe34bcc5ede62fd6f2fcba Oct 30 18:07:59.534105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.534023 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" event={"ID":"29abc014-6eb3-474b-95b4-cc26370fd325","Type":"ContainerStarted","Data":"47e255e1629efda2b3082ebddc8e9d408ceb850e51fe34bcc5ede62fd6f2fcba"} Oct 30 18:07:59.834775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.832937 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg"] Oct 30 18:07:59.840277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.840249 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:07:59.851813 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.851750 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg"] Oct 30 18:07:59.918693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.918658 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ef773b-33c1-4db5-b541-ecee76e8bba3-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:07:59.918842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:07:59.918741 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hf2g\" (UniqueName: \"kubernetes.io/projected/18ef773b-33c1-4db5-b541-ecee76e8bba3-kube-api-access-7hf2g\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.020219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.019962 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ef773b-33c1-4db5-b541-ecee76e8bba3-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.020219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.020075 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hf2g\" (UniqueName: \"kubernetes.io/projected/18ef773b-33c1-4db5-b541-ecee76e8bba3-kube-api-access-7hf2g\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.031263 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.031229 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18ef773b-33c1-4db5-b541-ecee76e8bba3-cert\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.038491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.038456 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hf2g\" (UniqueName: \"kubernetes.io/projected/18ef773b-33c1-4db5-b541-ecee76e8bba3-kube-api-access-7hf2g\") pod \"llmisvc-controller-manager-6466c554d8-rsmrg\" (UID: \"18ef773b-33c1-4db5-b541-ecee76e8bba3\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.161203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.161176 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:00.474066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.473980 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg"] Oct 30 18:08:00.477569 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:08:00.477538 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ef773b_33c1_4db5_b541_ecee76e8bba3.slice/crio-7d4ad8d45411211445a3c0df1fcb88a0dcf948dddecf3e740aeacac655b98ad4 WatchSource:0}: Error finding container 7d4ad8d45411211445a3c0df1fcb88a0dcf948dddecf3e740aeacac655b98ad4: Status 404 returned error can't find the container with id 7d4ad8d45411211445a3c0df1fcb88a0dcf948dddecf3e740aeacac655b98ad4 Oct 30 18:08:00.541569 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.541531 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" event={"ID":"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6","Type":"ContainerStarted","Data":"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4"} Oct 30 18:08:00.542084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.541742 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:08:00.543128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.543088 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" event={"ID":"18ef773b-33c1-4db5-b541-ecee76e8bba3","Type":"ContainerStarted","Data":"7d4ad8d45411211445a3c0df1fcb88a0dcf948dddecf3e740aeacac655b98ad4"} Oct 30 18:08:00.579677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:00.579613 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" podStartSLOduration=2.161762146 podStartE2EDuration="4.578657037s" totalImagesPullingTime="2.416894891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:00.576832674 +0000 UTC m=+431.397544451" watchObservedRunningTime="2026-10-30 18:08:00.578657037 +0000 UTC m=+431.399368824" Oct 30 18:08:01.552354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:01.552295 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" event={"ID":"18ef773b-33c1-4db5-b541-ecee76e8bba3","Type":"ContainerStarted","Data":"041fb36ec0a834890da933061bec9f573b6e48ede33183deebfb9858bd4e43a4"} Oct 30 18:08:01.587798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:01.587739 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" podStartSLOduration=2.24130603 podStartE2EDuration="2.587718125s" totalImagesPullingTime="346.412095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:01.586681032 +0000 UTC m=+432.407392807" watchObservedRunningTime="2026-10-30 18:08:01.587718125 +0000 UTC m=+432.408429901" Oct 30 18:08:01.645786 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:01.645756 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:01.645888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:01.645824 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:01.645888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:01.645857 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:02.574394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:02.574353 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" event={"ID":"29abc014-6eb3-474b-95b4-cc26370fd325","Type":"ContainerStarted","Data":"af0f35e0e2f5299dd8c931322ec41da5c008de9dc2a02d52d83a686eb9a448ca"} Oct 30 18:08:02.575155 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:02.575131 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:02.629930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:02.629858 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podStartSLOduration=1.738830873 podStartE2EDuration="4.629844366s" totalImagesPullingTime="2.891013493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:02.619717163 +0000 UTC m=+433.440428938" watchObservedRunningTime="2026-10-30 18:08:02.629844366 +0000 UTC m=+433.450556144" Oct 30 18:08:03.407023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:03.406976 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:08:03.408717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:03.408687 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hn66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:08:03.408870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:03.408741 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podUID="29abc014-6eb3-474b-95b4-cc26370fd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 18:08:04.407082 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:04.407041 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hn66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:08:04.407579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:04.407107 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podUID="29abc014-6eb3-474b-95b4-cc26370fd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 18:08:05.407438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:05.407399 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hn66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:08:05.407923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:05.407457 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podUID="29abc014-6eb3-474b-95b4-cc26370fd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 18:08:06.407646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:06.407572 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hn66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.408064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:06.407678 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podUID="29abc014-6eb3-474b-95b4-cc26370fd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 18:08:06.572803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:06.572755 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:08:07.407007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:07.406961 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-hn66q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.407200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:07.407030 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" podUID="29abc014-6eb3-474b-95b4-cc26370fd325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 30 18:08:08.412128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:08.412078 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:08:08.412682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:08.412559 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:08:08.413380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:08.413361 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-hn66q" Oct 30 18:08:10.456879 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:08:10.456843 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:08:28.190391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.190351 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k"] Oct 30 18:08:28.195378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.195355 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.214833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.214800 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k"] Oct 30 18:08:28.310091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310049 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310097 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8029cccf-1a8c-431e-a244-c840c94bb846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310181 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310221 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8029cccf-1a8c-431e-a244-c840c94bb846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310250 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310330 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4znr\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-kube-api-access-d4znr\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310358 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310382 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.310504 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.310463 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411517 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411572 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8029cccf-1a8c-431e-a244-c840c94bb846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411757 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411827 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4znr\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-kube-api-access-d4znr\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411859 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.411989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.411930 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412045 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412029 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412245 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412322 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412370 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412394 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8029cccf-1a8c-431e-a244-c840c94bb846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412399 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8029cccf-1a8c-431e-a244-c840c94bb846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.412832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.412642 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.414584 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.414565 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8029cccf-1a8c-431e-a244-c840c94bb846-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.414897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.414882 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8029cccf-1a8c-431e-a244-c840c94bb846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.429569 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.429538 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.429783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.429759 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4znr\" (UniqueName: \"kubernetes.io/projected/8029cccf-1a8c-431e-a244-c840c94bb846-kube-api-access-d4znr\") pod \"maas-default-gateway-openshift-default-77f556d647-n789k\" (UID: \"8029cccf-1a8c-431e-a244-c840c94bb846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.508802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.508706 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:28.825076 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.824996 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k"] Oct 30 18:08:28.828969 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:08:28.828944 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8029cccf_1a8c_431e_a244_c840c94bb846.slice/crio-11bf7ab97dc2b70682104a3ec55cb0818b262f1a3b668e6dbb4d0cf7deab17e3 WatchSource:0}: Error finding container 11bf7ab97dc2b70682104a3ec55cb0818b262f1a3b668e6dbb4d0cf7deab17e3: Status 404 returned error can't find the container with id 11bf7ab97dc2b70682104a3ec55cb0818b262f1a3b668e6dbb4d0cf7deab17e3 Oct 30 18:08:28.831126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.831094 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:28.831201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.831166 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:28.831201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.831196 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:28.961065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.961032 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" event={"ID":"8029cccf-1a8c-431e-a244-c840c94bb846","Type":"ContainerStarted","Data":"57f5173553869917f47840b9f83d2f85f7b6be3bda0c082a6494276a93f5ff46"} Oct 30 18:08:28.961253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.961091 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" event={"ID":"8029cccf-1a8c-431e-a244-c840c94bb846","Type":"ContainerStarted","Data":"11bf7ab97dc2b70682104a3ec55cb0818b262f1a3b668e6dbb4d0cf7deab17e3"} Oct 30 18:08:28.999517 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:28.999454 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podStartSLOduration=0.99943205 podStartE2EDuration="999.43205ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:28.99765895 +0000 UTC m=+459.818370744" watchObservedRunningTime="2026-10-30 18:08:28.99943205 +0000 UTC m=+459.820143822" Oct 30 18:08:29.509004 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:29.508966 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:29.510640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:29.510617 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.510713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:29.510666 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:30.509980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:30.509936 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.510474 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:30.510005 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:31.509713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:31.509666 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.509935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:31.509727 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:32.509717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:32.509684 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.510154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:32.509751 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:33.509646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.509583 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:33.509933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.509681 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:33.580260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.580221 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rsmrg" Oct 30 18:08:33.661631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.656289 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:08:33.674706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.656560 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" podUID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" containerName="manager" containerID="cri-o://4038a90f9932adcd5a85415626c00303a32ba010cf43bd41ba899211278a2df6" gracePeriod=30 Oct 30 18:08:33.997614 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.997563 5024 generic.go:356] "Generic (PLEG): container finished" podID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" containerID="4038a90f9932adcd5a85415626c00303a32ba010cf43bd41ba899211278a2df6" exitCode=0 Oct 30 18:08:33.997764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.997613 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" event={"ID":"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98","Type":"ContainerDied","Data":"4038a90f9932adcd5a85415626c00303a32ba010cf43bd41ba899211278a2df6"} Oct 30 18:08:33.997764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.997673 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" event={"ID":"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98","Type":"ContainerDied","Data":"580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6"} Oct 30 18:08:33.997764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:33.997693 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="580c2b2fda12adcb3dbeedb1dfcff2becd68c2bd458e2666d841112c052674a6" Oct 30 18:08:34.018159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.018122 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:08:34.088885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.088804 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\" (UniqueName: \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\") pod \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " Oct 30 18:08:34.089064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.088913 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\" (UniqueName: \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\") pod \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\" (UID: \"4dbe9c2a-9caf-483f-ac4a-0b20a3304a98\") " Oct 30 18:08:34.091323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.091289 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn" pod "4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" (UID: "4dbe9c2a-9caf-483f-ac4a-0b20a3304a98"). InnerVolumeSpecName "kube-api-access-zvghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:34.091323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.091299 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert" pod "4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" (UID: "4dbe9c2a-9caf-483f-ac4a-0b20a3304a98"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:34.189713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.189670 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvghn\" (UniqueName: \"kubernetes.io/projected/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-kube-api-access-zvghn\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:08:34.189713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.189705 5024 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:08:34.509795 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.509760 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n789k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:08:34.510002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:34.509813 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" podUID="8029cccf-1a8c-431e-a244-c840c94bb846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 18:08:35.002303 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.002270 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt" Oct 30 18:08:35.047811 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.047778 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:08:35.056811 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.056782 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-g46qt"] Oct 30 18:08:35.305655 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.305566 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" path="/var/lib/kubelet/pods/4dbe9c2a-9caf-483f-ac4a-0b20a3304a98/volumes" Oct 30 18:08:35.513704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.513668 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:35.514094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.513943 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:35.514697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:08:35.514677 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n789k" Oct 30 18:08:40.458823 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:08:40.458786 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:09:10.472398 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:09:10.472258 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:09:13.327965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.327922 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt"] Oct 30 18:09:13.328545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.328519 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" containerName="manager" Oct 30 18:09:13.328633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.328548 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" containerName="manager" Oct 30 18:09:13.328724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.328649 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbe9c2a-9caf-483f-ac4a-0b20a3304a98" containerName="manager" Oct 30 18:09:13.332466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.332441 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.356970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.356933 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt"] Oct 30 18:09:13.361968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.361929 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9475a6a1-002e-49b3-bb80-67db243d4e0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.361989 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362017 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362047 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362115 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcml\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-kube-api-access-hlcml\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362166 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362202 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362336 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.362585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.362368 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.462904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.462867 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.462904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.462907 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.462963 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9475a6a1-002e-49b3-bb80-67db243d4e0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.462993 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463170 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463247 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcml\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-kube-api-access-hlcml\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463283 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463310 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463319 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463322 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463559 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463743 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9475a6a1-002e-49b3-bb80-67db243d4e0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.463835 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.463804 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.465831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.465809 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.466013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.465994 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.481386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.481350 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.481631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.481575 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcml\" (UniqueName: \"kubernetes.io/projected/9475a6a1-002e-49b3-bb80-67db243d4e0f-kube-api-access-hlcml\") pod \"maas-default-gateway-openshift-default-77f556d647-vftgt\" (UID: \"9475a6a1-002e-49b3-bb80-67db243d4e0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.649151 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.649104 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:13.971872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.971794 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt"] Oct 30 18:09:13.975685 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:09:13.975655 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9475a6a1_002e_49b3_bb80_67db243d4e0f.slice/crio-958e59bd6a50ca02e8090f5c82e3c85ca46a73418c204546ef920d2be4070b2e WatchSource:0}: Error finding container 958e59bd6a50ca02e8090f5c82e3c85ca46a73418c204546ef920d2be4070b2e: Status 404 returned error can't find the container with id 958e59bd6a50ca02e8090f5c82e3c85ca46a73418c204546ef920d2be4070b2e Oct 30 18:09:13.977896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.977863 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:13.977991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.977939 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:13.977991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:13.977970 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:14.227647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:14.227520 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" event={"ID":"9475a6a1-002e-49b3-bb80-67db243d4e0f","Type":"ContainerStarted","Data":"ecfe795709a1cd3f2f9181a0d1b700cbccc4683f0e42f3df1192c28e0ff73f15"} Oct 30 18:09:14.227647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:14.227628 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" event={"ID":"9475a6a1-002e-49b3-bb80-67db243d4e0f","Type":"ContainerStarted","Data":"958e59bd6a50ca02e8090f5c82e3c85ca46a73418c204546ef920d2be4070b2e"} Oct 30 18:09:14.274633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:14.272056 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podStartSLOduration=1.272034489 podStartE2EDuration="1.272034489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:14.271313913 +0000 UTC m=+505.092025692" watchObservedRunningTime="2026-10-30 18:09:14.272034489 +0000 UTC m=+505.092746291" Oct 30 18:09:14.649333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:14.649276 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:15.653875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:15.653798 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.654577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:15.654535 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:16.650635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:16.650317 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.650635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:16.650388 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:17.656752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:17.656713 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.657348 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:17.657316 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:18.650054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:18.650012 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:18.650342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:18.650313 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:19.650622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:19.650043 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:19.650622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:19.650112 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:20.649748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:20.649710 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vftgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 18:09:20.649936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:20.649768 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" podUID="9475a6a1-002e-49b3-bb80-67db243d4e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 18:09:22.204798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:22.204763 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:22.205277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:22.205120 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:22.206094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:09:22.206074 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vftgt" Oct 30 18:09:40.457844 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:09:40.457801 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:10:10.456085 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:10:10.455979 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:10:33.417485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.417393 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:10:33.420746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.420720 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.441490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.441456 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-fb2fa0-serving-cert\"" Oct 30 18:10:33.455772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.455738 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:10:33.553834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.553804 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.554037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.553845 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.554037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.553879 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.554037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.553984 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8688z\" (UniqueName: \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.655284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.655248 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.655532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.655301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.655532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.655346 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.655532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.655384 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8688z\" (UniqueName: \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.656088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.656062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.657971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.657944 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.658178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.658154 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.680948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.680868 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8688z\" (UniqueName: \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\") pod \"maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:33.733385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:33.733341 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:34.037583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:34.037506 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:10:34.041993 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:10:34.041967 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4583e34d_8f75_4ab2_9ae4_f0a542175d1c.slice/crio-d34cf1fafc58f48799df355defa5954fbb7195d9b2e552beb0adf8a294cd7102 WatchSource:0}: Error finding container d34cf1fafc58f48799df355defa5954fbb7195d9b2e552beb0adf8a294cd7102: Status 404 returned error can't find the container with id d34cf1fafc58f48799df355defa5954fbb7195d9b2e552beb0adf8a294cd7102 Oct 30 18:10:34.661889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:34.661842 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" event={"ID":"4583e34d-8f75-4ab2-9ae4-f0a542175d1c","Type":"ContainerStarted","Data":"0a58a739d61f38317dfc16cc900ee14dae1c2c08fcf8736dd6201d59f2a7e23e"} Oct 30 18:10:34.662350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:34.661906 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" event={"ID":"4583e34d-8f75-4ab2-9ae4-f0a542175d1c","Type":"ContainerStarted","Data":"d34cf1fafc58f48799df355defa5954fbb7195d9b2e552beb0adf8a294cd7102"} Oct 30 18:10:34.662350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:34.661981 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:34.697743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:34.697614 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" podStartSLOduration=1.302459325 podStartE2EDuration="1.697578994s" totalImagesPullingTime="395.119669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:34.696031515 +0000 UTC m=+585.516743291" watchObservedRunningTime="2026-10-30 18:10:34.697578994 +0000 UTC m=+585.518290774" Oct 30 18:10:35.138677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.138578 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:10:35.142368 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.142345 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.172640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.164993 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fdf4da-serving-cert\"" Oct 30 18:10:35.172640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.168799 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:10:35.272267 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.272227 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.272267 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.272266 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.272494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.272352 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.272494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.272441 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.373391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.373336 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.373592 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.373395 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.373592 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.373494 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.373744 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.373578 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.374208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.374178 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.376243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.376220 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.376379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.376356 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.391644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.391576 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\") pod \"maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.456576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.456531 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:35.762469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:35.762378 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:10:35.766535 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:10:35.766503 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bfc68b_7fca_4862_9557_ac06eb693fb0.slice/crio-4b42da46bdd16831308059ea1a4160eecc1bec0b4cbff40326f974c0829ae22e WatchSource:0}: Error finding container 4b42da46bdd16831308059ea1a4160eecc1bec0b4cbff40326f974c0829ae22e: Status 404 returned error can't find the container with id 4b42da46bdd16831308059ea1a4160eecc1bec0b4cbff40326f974c0829ae22e Oct 30 18:10:36.673675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:36.673638 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" event={"ID":"c4bfc68b-7fca-4862-9557-ac06eb693fb0","Type":"ContainerStarted","Data":"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4"} Oct 30 18:10:36.673871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:36.673691 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" event={"ID":"c4bfc68b-7fca-4862-9557-ac06eb693fb0","Type":"ContainerStarted","Data":"4b42da46bdd16831308059ea1a4160eecc1bec0b4cbff40326f974c0829ae22e"} Oct 30 18:10:36.673871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:36.673780 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:36.739988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:36.739929 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" podStartSLOduration=1.320955436 podStartE2EDuration="1.73991345s" totalImagesPullingTime="418.958014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:36.738897979 +0000 UTC m=+587.559609756" watchObservedRunningTime="2026-10-30 18:10:36.73991345 +0000 UTC m=+587.560625226" Oct 30 18:10:38.754690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.754649 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:10:38.758137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.758115 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.782642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.782578 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8b6cf1-serving-cert\"" Oct 30 18:10:38.786104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.786070 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:10:38.809271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.809225 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.809271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.809268 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.809530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.809294 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.809530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.809463 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.910490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.910443 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.910731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.910505 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.910731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.910545 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.910731 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:10:38.910639 5024 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-serving-cert: secret "maas-api-e2e-worker-w3-8b6cf1-serving-cert" not found Oct 30 18:10:38.910731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.910687 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.910731 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:10:38.910716 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls podName:86ee8676-bb5c-455e-9456-4b1d54381b92 nodeName:}" failed. No retries permitted until 2026-10-30 18:10:39.410694919 +0000 UTC m=+590.231406691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls") pod "maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" (UID: "86ee8676-bb5c-455e-9456-4b1d54381b92") : secret "maas-api-e2e-worker-w3-8b6cf1-serving-cert" not found Oct 30 18:10:38.911421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.911393 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.913220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.913193 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:38.931806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:38.931769 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:39.414565 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:39.414528 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:39.417228 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:39.417203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:39.672515 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:39.672413 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:39.979062 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:39.978958 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:10:39.982981 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:10:39.982952 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ee8676_bb5c_455e_9456_4b1d54381b92.slice/crio-870332b3881447ee17ce64c3cea321dc07dffb7afffa5d0e09a53028d6e9bc19 WatchSource:0}: Error finding container 870332b3881447ee17ce64c3cea321dc07dffb7afffa5d0e09a53028d6e9bc19: Status 404 returned error can't find the container with id 870332b3881447ee17ce64c3cea321dc07dffb7afffa5d0e09a53028d6e9bc19 Oct 30 18:10:40.460033 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:10:40.459977 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:10:40.672473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:40.672443 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:10:40.702724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:40.702678 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" event={"ID":"86ee8676-bb5c-455e-9456-4b1d54381b92","Type":"ContainerStarted","Data":"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf"} Oct 30 18:10:40.702923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:40.702749 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" event={"ID":"86ee8676-bb5c-455e-9456-4b1d54381b92","Type":"ContainerStarted","Data":"870332b3881447ee17ce64c3cea321dc07dffb7afffa5d0e09a53028d6e9bc19"} Oct 30 18:10:40.702923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:40.702772 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:40.752681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:40.752618 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" podStartSLOduration=2.422515265 podStartE2EDuration="2.752566219s" totalImagesPullingTime="330.050954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:40.750469121 +0000 UTC m=+591.571180897" watchObservedRunningTime="2026-10-30 18:10:40.752566219 +0000 UTC m=+591.573277993" Oct 30 18:10:42.358630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.358575 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:10:42.362430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.362413 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.378014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.377982 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ef13cc-serving-cert\"" Oct 30 18:10:42.391033 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.391002 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:10:42.444365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.444335 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.444365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.444378 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6tw8\" (UniqueName: \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.444649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.444555 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.444649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.444620 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.546063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.546030 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.546226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.546077 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.546226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.546108 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.546226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.546134 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6tw8\" (UniqueName: \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.547019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.546993 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.549089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.549060 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.553064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.553022 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.564657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.564621 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6tw8\" (UniqueName: \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\") pod \"maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.675734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.675688 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:42.684463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.684435 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:10:42.983322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:42.983240 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:10:42.986780 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:10:42.986751 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94164130_02ad_4a1c_8280_db911430ad97.slice/crio-42a047253bd4ba6b485b47220407ec25b0385f1917ef8bdadf155c8d0e14cb9f WatchSource:0}: Error finding container 42a047253bd4ba6b485b47220407ec25b0385f1917ef8bdadf155c8d0e14cb9f: Status 404 returned error can't find the container with id 42a047253bd4ba6b485b47220407ec25b0385f1917ef8bdadf155c8d0e14cb9f Oct 30 18:10:43.722119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:43.722077 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" event={"ID":"94164130-02ad-4a1c-8280-db911430ad97","Type":"ContainerStarted","Data":"1efb4dfe620da7933777b6b3d43690595bfcace4b38f3ce98b5cdebe86699e4b"} Oct 30 18:10:43.722583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:43.722133 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" event={"ID":"94164130-02ad-4a1c-8280-db911430ad97","Type":"ContainerStarted","Data":"42a047253bd4ba6b485b47220407ec25b0385f1917ef8bdadf155c8d0e14cb9f"} Oct 30 18:10:43.722583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:43.722154 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:43.760641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:43.760566 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" podStartSLOduration=1.385727108 podStartE2EDuration="1.760550498s" totalImagesPullingTime="374.82339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:43.760541751 +0000 UTC m=+594.581253526" watchObservedRunningTime="2026-10-30 18:10:43.760550498 +0000 UTC m=+594.581262271" Oct 30 18:10:45.827348 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.827312 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:10:45.831223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.831204 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.855206 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.855171 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ef6ac-serving-cert\"" Oct 30 18:10:45.858899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.858552 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:10:45.879633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.879574 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.879815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.879717 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.879815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.879794 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvzv\" (UniqueName: \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.879900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.879890 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.980683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.980629 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.980899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.980730 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.980899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.980766 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.980899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.980806 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvzv\" (UniqueName: \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.981384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.981359 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.983525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.983499 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.983656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.983497 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:45.998139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:45.998105 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvzv\" (UniqueName: \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\") pod \"maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:46.144980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:46.144939 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:46.461876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:46.461840 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:10:46.463920 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:10:46.463892 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b54f788_e7e4_439f_98b1_2c7ba3049db0.slice/crio-48c64f56316b415b04a8a0a27bc44431c42ac6663bc40afe6b1c25b936d5cde3 WatchSource:0}: Error finding container 48c64f56316b415b04a8a0a27bc44431c42ac6663bc40afe6b1c25b936d5cde3: Status 404 returned error can't find the container with id 48c64f56316b415b04a8a0a27bc44431c42ac6663bc40afe6b1c25b936d5cde3 Oct 30 18:10:46.713992 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:46.713910 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:10:46.743029 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:46.742992 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" event={"ID":"2b54f788-e7e4-439f-98b1-2c7ba3049db0","Type":"ContainerStarted","Data":"48c64f56316b415b04a8a0a27bc44431c42ac6663bc40afe6b1c25b936d5cde3"} Oct 30 18:10:47.750996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:47.750957 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" event={"ID":"2b54f788-e7e4-439f-98b1-2c7ba3049db0","Type":"ContainerStarted","Data":"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd"} Oct 30 18:10:47.751406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:47.751083 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:10:47.789467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:47.789299 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" podStartSLOduration=2.424324448 podStartE2EDuration="2.789280363s" totalImagesPullingTime="364.955915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:47.788548818 +0000 UTC m=+598.609260595" watchObservedRunningTime="2026-10-30 18:10:47.789280363 +0000 UTC m=+598.609992140" Oct 30 18:10:49.369305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:49.369269 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:10:49.369780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:49.369269 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:10:49.732905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:49.732877 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:10:53.762480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:10:53.762450 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:11:08.916579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:08.916531 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb"] Oct 30 18:11:08.921064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:08.921037 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:08.950493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:08.950452 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb"] Oct 30 18:11:09.020189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020155 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020197 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020226 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020409 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020527 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9smd\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020581 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020709 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.020761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.020728 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.033517 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.033481 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" Oct 30 18:11:09.121560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121521 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.121560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121563 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.121834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121584 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.121834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121752 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.121834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121811 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121848 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.121908 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122007 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9smd\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122055 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122116 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122122 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122221 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122621 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.122805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.122621 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.124367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.124342 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.124710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.124685 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.139288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.139260 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.139551 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.139527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9smd\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.237302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.237198 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:09.563354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.563246 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:11:09.567955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.567929 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.579897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.579864 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb"] Oct 30 18:11:09.584650 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:09.584586 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa94489_c8bd_41ef_80de_2578bc7b000e.slice/crio-1b8606e73f04fd2e8c09d8dd5c61606b3ab189c5757568a9343eb52409843c69 WatchSource:0}: Error finding container 1b8606e73f04fd2e8c09d8dd5c61606b3ab189c5757568a9343eb52409843c69: Status 404 returned error can't find the container with id 1b8606e73f04fd2e8c09d8dd5c61606b3ab189c5757568a9343eb52409843c69 Oct 30 18:11:09.587126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.587093 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:09.587241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.587179 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:09.587241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.587222 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:09.602578 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.602539 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" Oct 30 18:11:09.604298 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.604275 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:11:09.611533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.611459 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" Oct 30 18:11:09.622728 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.611568 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" Oct 30 18:11:09.622728 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.611783 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"simulator-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:11:09.729581 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729528 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.729838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729585 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.729838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729664 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdwm\" (UniqueName: \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.729838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729692 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.729838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729735 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.729838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.729769 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.830995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.830890 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.830995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.830953 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.830995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.830987 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdwm\" (UniqueName: \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.831299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.831019 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.831299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.831081 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.831299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.831120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.831952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.831924 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.832078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.831987 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.832364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.832342 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.834586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.834555 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.836272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.836244 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.852107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.852077 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdwm\" (UniqueName: \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\") pod \"simulator-w5-fdf4da-kserve-9996bf9df-6w75m\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.882681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.882640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" event={"ID":"3fa94489-c8bd-41ef-80de-2578bc7b000e","Type":"ContainerStarted","Data":"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738"} Oct 30 18:11:09.882868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.882711 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" event={"ID":"3fa94489-c8bd-41ef-80de-2578bc7b000e","Type":"ContainerStarted","Data":"1b8606e73f04fd2e8c09d8dd5c61606b3ab189c5757568a9343eb52409843c69"} Oct 30 18:11:09.888809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.888766 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:09.931630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:09.931536 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podStartSLOduration=1.93152005 podStartE2EDuration="1.93152005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:09.928968069 +0000 UTC m=+620.749679841" watchObservedRunningTime="2026-10-30 18:11:09.93152005 +0000 UTC m=+620.752231827" Oct 30 18:11:10.197214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.197180 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:11:10.200994 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:10.200961 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bff7d2b_a767_434f_83c8_aa847746f21c.slice/crio-9d9738b208133a9e18f341ccb291ab828c2fb77ce8135eb2b9d27dce5d7e8b08 WatchSource:0}: Error finding container 9d9738b208133a9e18f341ccb291ab828c2fb77ce8135eb2b9d27dce5d7e8b08: Status 404 returned error can't find the container with id 9d9738b208133a9e18f341ccb291ab828c2fb77ce8135eb2b9d27dce5d7e8b08 Oct 30 18:11:10.203211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.203191 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:11:10.238428 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.238394 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:10.239844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.239814 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:10.239978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.239867 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:10.493350 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:11:10.493257 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:11:10.898972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:10.898936 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" event={"ID":"5bff7d2b-a767-434f-83c8-aa847746f21c","Type":"ContainerStarted","Data":"9d9738b208133a9e18f341ccb291ab828c2fb77ce8135eb2b9d27dce5d7e8b08"} Oct 30 18:11:11.238356 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.238270 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.238356 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.238343 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:11.881974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.881930 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:11:11.897316 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.888522 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.908189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.907360 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"simulator-w3-8b6cf1-kserve-self-signed-certs\"" Oct 30 18:11:11.908189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.907470 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"default-dockercfg-jl62g\"" Oct 30 18:11:11.908189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.908075 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:11:11.912633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.911718 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"kube-root-ca.crt\"" Oct 30 18:11:11.912633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.911781 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"openshift-service-ca.crt\"" Oct 30 18:11:11.971624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971435 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6t78\" (UniqueName: \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.971624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971538 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.971624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971571 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.971974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971644 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.971974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971673 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:11.971974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:11.971773 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.077241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.077023 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6t78\" (UniqueName: \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.078790 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.079320 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.079246 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.079708 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.079763 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.080025 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.080084 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.081658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.080683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.087014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.086979 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.087382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.087361 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.097806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.097766 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6t78\" (UniqueName: \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\") pod \"simulator-w3-8b6cf1-kserve-678594f54b-m5nh2\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.212655 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.212446 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:12.237933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.237901 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.238049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.237966 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:12.597188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.597109 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:11:12.600969 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:12.600939 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8396c956_04cc_4dc0_8abd_8103d8ef1ca4.slice/crio-91f9c0c4daab7ccfa77564125d9d32c42565dfe6f886c91628dc9248b5e4c37e WatchSource:0}: Error finding container 91f9c0c4daab7ccfa77564125d9d32c42565dfe6f886c91628dc9248b5e4c37e: Status 404 returned error can't find the container with id 91f9c0c4daab7ccfa77564125d9d32c42565dfe6f886c91628dc9248b5e4c37e Oct 30 18:11:12.919521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.919480 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" event={"ID":"5bff7d2b-a767-434f-83c8-aa847746f21c","Type":"ContainerStarted","Data":"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8"} Oct 30 18:11:12.921329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.921298 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" event={"ID":"8396c956-04cc-4dc0-8abd-8103d8ef1ca4","Type":"ContainerStarted","Data":"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79"} Oct 30 18:11:12.921329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.921335 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" event={"ID":"8396c956-04cc-4dc0-8abd-8103d8ef1ca4","Type":"ContainerStarted","Data":"91f9c0c4daab7ccfa77564125d9d32c42565dfe6f886c91628dc9248b5e4c37e"} Oct 30 18:11:12.976481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:12.976406 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podStartSLOduration=1.94176996 podStartE2EDuration="3.976384743s" totalImagesPullingTime="2.034614783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:12.975330769 +0000 UTC m=+623.796042572" watchObservedRunningTime="2026-10-30 18:11:12.976384743 +0000 UTC m=+623.797096520" Oct 30 18:11:13.009825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:13.009754 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podStartSLOduration=2.009736859 podStartE2EDuration="2.009736859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:13.008046765 +0000 UTC m=+623.828758543" watchObservedRunningTime="2026-10-30 18:11:13.009736859 +0000 UTC m=+623.830448636" Oct 30 18:11:13.213083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:13.212976 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:13.237815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:13.237782 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.238025 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:13.237835 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:13.889945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:13.889908 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:14.238105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:14.238012 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.238105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:14.238078 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:15.237801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:15.237768 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.238309 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:15.237827 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 18:11:16.246050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:16.246017 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:16.246554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:16.246353 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:16.247417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:16.247258 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:11:19.889468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:19.889431 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:19.902953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:19.902925 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:19.979388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:19.979359 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:11:22.213678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:22.213640 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:22.226729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:22.226696 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:22.997428 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:22.997391 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:11:37.607861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.607824 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:11:37.612063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.612038 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.629829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.629792 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:11:37.636278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.636244 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" Oct 30 18:11:37.636447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.636405 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" Oct 30 18:11:37.636536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.636515 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"premium-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:11:37.636638 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.636587 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" Oct 30 18:11:37.756364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756324 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.756615 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756388 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.756615 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756528 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf86t\" (UniqueName: \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.756742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756616 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.756742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756675 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.756860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.756744 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857496 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857440 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857525 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857585 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857676 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.857861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.857717 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf86t\" (UniqueName: \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.858084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.858021 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.858084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.858033 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.858207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.858116 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.861026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.861004 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.861489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.861465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.880689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.880653 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf86t\" (UniqueName: \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\") pod \"premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:37.926331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:37.926277 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:38.229005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:38.228968 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:11:38.233334 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:38.233303 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38680810_b7c9_485b_87eb_b5c34b9d1436.slice/crio-645cdcee0233842d6c1dbeab4254ad13234985fadbfcbe146f90ea1fe3d18f9a WatchSource:0}: Error finding container 645cdcee0233842d6c1dbeab4254ad13234985fadbfcbe146f90ea1fe3d18f9a: Status 404 returned error can't find the container with id 645cdcee0233842d6c1dbeab4254ad13234985fadbfcbe146f90ea1fe3d18f9a Oct 30 18:11:39.085740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:39.085694 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" event={"ID":"38680810-b7c9-485b-87eb-b5c34b9d1436","Type":"ContainerStarted","Data":"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe"} Oct 30 18:11:39.086178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:39.085750 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" event={"ID":"38680810-b7c9-485b-87eb-b5c34b9d1436","Type":"ContainerStarted","Data":"645cdcee0233842d6c1dbeab4254ad13234985fadbfcbe146f90ea1fe3d18f9a"} Oct 30 18:11:39.122256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:39.122187 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podStartSLOduration=2.122167353 podStartE2EDuration="2.122167353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:39.120770711 +0000 UTC m=+649.941482466" watchObservedRunningTime="2026-10-30 18:11:39.122167353 +0000 UTC m=+649.942879133" Oct 30 18:11:39.926535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:39.926496 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:40.458737 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:11:40.458698 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:11:47.926672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:47.926635 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:47.939945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:47.939917 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:48.148776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:48.148742 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:11:49.391627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.391576 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:11:49.397900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.397871 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.418019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.417989 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:11:49.421002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.420980 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"premium-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:11:49.580942 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.580901 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.581234 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.581213 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.581376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.581361 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.581497 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.581482 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlzh\" (UniqueName: \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.581705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.581687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.581881 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.581856 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683073 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683149 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683205 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683257 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlzh\" (UniqueName: \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683344 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683375 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683526 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683655 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.683788 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.683763 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.685661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.685637 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.686248 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.686228 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.700667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.700640 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlzh\" (UniqueName: \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\") pod \"premium-w5-fdf4da-kserve-8577fffc6d-gkfzv\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:49.718618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:49.718564 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:50.023086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:50.023050 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:11:50.025818 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:50.025790 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd367b5f9_6441_46e7_968c_11034679ef97.slice/crio-291c9ab400a6e5024bcda2146ae652173fdf1fe58b2ffadedf3915b6e1538dd1 WatchSource:0}: Error finding container 291c9ab400a6e5024bcda2146ae652173fdf1fe58b2ffadedf3915b6e1538dd1: Status 404 returned error can't find the container with id 291c9ab400a6e5024bcda2146ae652173fdf1fe58b2ffadedf3915b6e1538dd1 Oct 30 18:11:50.159448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:50.159394 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" event={"ID":"d367b5f9-6441-46e7-968c-11034679ef97","Type":"ContainerStarted","Data":"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8"} Oct 30 18:11:50.159675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:50.159462 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" event={"ID":"d367b5f9-6441-46e7-968c-11034679ef97","Type":"ContainerStarted","Data":"291c9ab400a6e5024bcda2146ae652173fdf1fe58b2ffadedf3915b6e1538dd1"} Oct 30 18:11:50.194606 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:50.194549 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podStartSLOduration=1.194535777 podStartE2EDuration="1.194535777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:50.193689351 +0000 UTC m=+661.014401124" watchObservedRunningTime="2026-10-30 18:11:50.194535777 +0000 UTC m=+661.015247550" Oct 30 18:11:50.719207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:50.719159 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:54.040091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.040055 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:11:54.044402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.044381 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.064034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.064009 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:11:54.131480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131438 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131499 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131651 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131708 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131827 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.131949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131920 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhbq\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.132059 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.131997 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.132059 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.132029 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.232781 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.232734 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.232990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.232795 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.232990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.232834 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.232990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.232871 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.232990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.232939 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233027 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233111 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233166 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233196 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhbq\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233267 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233195 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233530 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233696 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.233849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.233771 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.235758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.235732 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.236111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.236084 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.250268 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.250235 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.250867 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.250838 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhbq\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.364584 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.364494 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:54.692125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.691924 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:11:54.695108 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:11:54.695081 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aac3789_e96f_4e9a_a95e_9fb527d0dee3.slice/crio-151c6245e452c894e513d840513474b09ae783610abd6dabf55c7f4342b66412 WatchSource:0}: Error finding container 151c6245e452c894e513d840513474b09ae783610abd6dabf55c7f4342b66412: Status 404 returned error can't find the container with id 151c6245e452c894e513d840513474b09ae783610abd6dabf55c7f4342b66412 Oct 30 18:11:54.697574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.697542 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:54.697676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.697628 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:54.697676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:54.697658 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:55.201043 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.201000 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" event={"ID":"6aac3789-e96f-4e9a-a95e-9fb527d0dee3","Type":"ContainerStarted","Data":"8e630cebf0637011ec974d177f7889e2c3f33f0c85d823ca6f159dbbe372251d"} Oct 30 18:11:55.201520 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.201082 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" event={"ID":"6aac3789-e96f-4e9a-a95e-9fb527d0dee3","Type":"ContainerStarted","Data":"151c6245e452c894e513d840513474b09ae783610abd6dabf55c7f4342b66412"} Oct 30 18:11:55.257858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.257803 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podStartSLOduration=1.257788761 podStartE2EDuration="1.257788761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:55.247535141 +0000 UTC m=+666.068246918" watchObservedRunningTime="2026-10-30 18:11:55.257788761 +0000 UTC m=+666.078500537" Oct 30 18:11:55.365671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.365630 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:11:55.367677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.367640 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:11:55.367830 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:55.367739 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:11:56.367624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:56.367233 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.367624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:56.367311 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:11:57.369622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:57.369033 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.369622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:57.369112 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:11:58.365676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:58.365061 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.365676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:58.365137 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:11:59.368625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.365028 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.368625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.365148 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:11:59.514395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.514353 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:11:59.518754 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.518727 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.535854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.535822 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:11:59.543901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.543868 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"premium-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:11:59.544126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.543927 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" Oct 30 18:11:59.544126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.543978 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" Oct 30 18:11:59.544126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.544046 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" Oct 30 18:11:59.601161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601131 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.601345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601236 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.601345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601291 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.601345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601327 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.601461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601346 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr47q\" (UniqueName: \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.601461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.601439 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.702904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.702861 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703003 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703318 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703292 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703401 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703380 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703099 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703694 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.703861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.703847 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr47q\" (UniqueName: \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.704467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.704442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.705841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.705818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.707633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.707574 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.719725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.718922 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:59.721764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.721734 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr47q\" (UniqueName: \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\") pod \"premium-w6-ef13cc-kserve-79bb7f864c-z8nwj\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:11:59.741286 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.740950 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:11:59.832765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:11:59.832721 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:12:00.180734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:00.178480 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:12:00.230551 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:00.230516 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" event={"ID":"9ac176aa-6e12-4e28-803c-55c3e925ce39","Type":"ContainerStarted","Data":"b6d28d8ac89ceb071db6106c47fb7ae8b4e5d7dc234e2c04db602a46f71f8f0b"} Oct 30 18:12:00.244331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:00.244303 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:12:00.366526 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:00.366063 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.366526 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:00.366126 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 18:12:01.242009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.241970 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" event={"ID":"9ac176aa-6e12-4e28-803c-55c3e925ce39","Type":"ContainerStarted","Data":"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1"} Oct 30 18:12:01.278987 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.278921 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podStartSLOduration=2.278899049 podStartE2EDuration="2.278899049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:01.275711269 +0000 UTC m=+672.096423048" watchObservedRunningTime="2026-10-30 18:12:01.278899049 +0000 UTC m=+672.099610826" Oct 30 18:12:01.439077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.439043 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:12:01.439514 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.439493 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:12:01.440383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.440362 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:12:01.833551 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:01.833508 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:12:09.833236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:09.833196 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:12:09.846364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:09.846332 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:12:10.304864 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:10.304830 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:12:10.455784 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:12:10.455744 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:12:23.763023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.762984 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:12:23.768305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.768283 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.786729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.786695 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:12:23.793938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.793898 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" Oct 30 18:12:23.876832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.876788 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.876832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.876836 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.876971 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877048 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877141 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877184 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877281 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877314 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.877408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.877366 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdf9v\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.978679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.978884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978725 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.978884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978857 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978910 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978951 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.978972 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979003 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdf9v\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979064 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979096 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979395 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979564 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.979900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.979877 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.981801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.981778 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.981889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.981810 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.996335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.996306 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdf9v\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:23.996490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:23.996311 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:24.082898 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:24.082812 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:24.402164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:24.402051 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:12:24.405210 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:12:24.405180 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae9bd74_c688_4ba5_929d_cbe2ec877509.slice/crio-4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0 WatchSource:0}: Error finding container 4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0: Status 404 returned error can't find the container with id 4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0 Oct 30 18:12:24.407653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:24.407621 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:24.407739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:24.407686 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:24.407739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:24.407721 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:25.377589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:25.377547 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" event={"ID":"1ae9bd74-c688-4ba5-929d-cbe2ec877509","Type":"ContainerStarted","Data":"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510"} Oct 30 18:12:25.378028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:25.377628 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" event={"ID":"1ae9bd74-c688-4ba5-929d-cbe2ec877509","Type":"ContainerStarted","Data":"4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0"} Oct 30 18:12:25.435307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:25.435247 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podStartSLOduration=2.43523174 podStartE2EDuration="2.43523174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:25.4230564 +0000 UTC m=+696.243768178" watchObservedRunningTime="2026-10-30 18:12:25.43523174 +0000 UTC m=+696.255943506" Oct 30 18:12:26.083887 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:26.083829 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:26.085746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:26.085709 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:12:26.093978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:26.093906 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:12:27.083776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:27.083736 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:12:27.084192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:27.083803 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:12:28.084100 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:28.084063 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:12:28.084671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:28.084133 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:12:29.084186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:29.084149 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:12:29.084583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:29.084218 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:12:30.084188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:30.084156 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:12:30.084378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:30.084209 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 18:12:31.928481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:31.928445 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:31.929409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:31.929347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:31.930892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:31.930872 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:12:38.348667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.348615 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:12:38.353066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.353043 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.366045 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.366016 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" Oct 30 18:12:38.380481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.380448 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:12:38.462591 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462549 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462591 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462607 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462709 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462798 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462850 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462897 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhxk\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.462988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462966 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.463022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.462996 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.463097 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.463078 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564181 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564141 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564196 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564266 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564299 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564331 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564372 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564448 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564508 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564544 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhxk\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564657 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564736 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564758 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.564878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.564813 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.565247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.565227 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.566930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.566908 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.567149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.567132 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.586576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.586536 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.586576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.586570 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhxk\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.666783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.666738 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:38.728885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.728856 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:12:38.733257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.733231 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.750877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.750846 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:12:38.763888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.763863 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" Oct 30 18:12:38.769245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769211 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769258 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769403 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769458 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769523 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769562 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769682 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.769850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.769738 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bchf\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871140 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bchf\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871200 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871226 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871317 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871372 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871404 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871443 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871501 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871533 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.871858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871809 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.872005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.871980 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.872102 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.872078 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.872265 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.872248 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.872393 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.872371 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.873741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.873725 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.874094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.874074 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.889457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.889426 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.889690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.889671 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bchf\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:38.983869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.983843 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:12:38.986665 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:12:38.986640 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3505cae6_b485_4d00_aaba_fe4655b66465.slice/crio-cdc86b84b4ecd5c96716807ad908dbc17f9c74f04cf55e8c0530def6610537e4 WatchSource:0}: Error finding container cdc86b84b4ecd5c96716807ad908dbc17f9c74f04cf55e8c0530def6610537e4: Status 404 returned error can't find the container with id cdc86b84b4ecd5c96716807ad908dbc17f9c74f04cf55e8c0530def6610537e4 Oct 30 18:12:38.989203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.989172 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:38.989275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.989235 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:38.989275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:38.989265 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:39.057427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.057403 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:39.377134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.377051 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:12:39.380899 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:12:39.380868 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ee49bb_4874_4b7c_832f_cccfddea8e9e.slice/crio-a34b4e20252e1ecfe6efd00244afc8df1382adacd80740d99b0c34a562e48a61 WatchSource:0}: Error finding container a34b4e20252e1ecfe6efd00244afc8df1382adacd80740d99b0c34a562e48a61: Status 404 returned error can't find the container with id a34b4e20252e1ecfe6efd00244afc8df1382adacd80740d99b0c34a562e48a61 Oct 30 18:12:39.383419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.383384 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:39.383525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.383453 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:39.383525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.383480 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:12:39.457483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.457445 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" event={"ID":"72ee49bb-4874-4b7c-832f-cccfddea8e9e","Type":"ContainerStarted","Data":"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63"} Oct 30 18:12:39.457721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.457509 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" event={"ID":"72ee49bb-4874-4b7c-832f-cccfddea8e9e","Type":"ContainerStarted","Data":"a34b4e20252e1ecfe6efd00244afc8df1382adacd80740d99b0c34a562e48a61"} Oct 30 18:12:39.459199 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.459173 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" event={"ID":"3505cae6-b485-4d00-aaba-fe4655b66465","Type":"ContainerStarted","Data":"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962"} Oct 30 18:12:39.459336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.459223 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" event={"ID":"3505cae6-b485-4d00-aaba-fe4655b66465","Type":"ContainerStarted","Data":"cdc86b84b4ecd5c96716807ad908dbc17f9c74f04cf55e8c0530def6610537e4"} Oct 30 18:12:39.500960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.500882 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podStartSLOduration=1.500864416 podStartE2EDuration="1.500864416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.491801261 +0000 UTC m=+710.312513038" watchObservedRunningTime="2026-10-30 18:12:39.500864416 +0000 UTC m=+710.321576192" Oct 30 18:12:39.529748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.529691 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podStartSLOduration=1.529671471 podStartE2EDuration="1.529671471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.527672866 +0000 UTC m=+710.348384645" watchObservedRunningTime="2026-10-30 18:12:39.529671471 +0000 UTC m=+710.350383285" Oct 30 18:12:39.668624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:39.667685 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:40.067764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:40.067667 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:40.072757 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:40.072711 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:40.072933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:40.072779 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:40.515656 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:12:40.515621 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:12:40.674760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:40.674720 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:40.675060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:40.675026 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:41.067078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:41.067032 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.067384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:41.067350 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:41.674073 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:41.674032 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.674755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:41.674720 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:42.059685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:42.058024 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:42.059685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:42.058092 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:42.679626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:42.677068 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:42.679626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:42.677128 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:43.063727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:43.063634 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:43.063983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:43.063957 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:43.704623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:43.702183 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:43.704623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:43.702248 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:44.063675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:44.063031 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.063675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:44.063097 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:44.667226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:44.667185 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.671741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:44.671691 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:45.059126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:45.059022 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:45.059745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:45.059710 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:45.676111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:45.676068 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:12:45.676426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:45.676386 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 18:12:46.059020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:46.058916 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.059020 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:46.058987 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 18:12:46.917745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:46.917712 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:46.918117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:46.918094 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:46.919347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:46.919326 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:12:47.063054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:47.063024 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:47.063559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:47.063544 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:12:47.064479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:12:47.064449 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:13:09.236790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.236745 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:13:09.240818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.240797 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.271200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.271150 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:13:09.390901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.390856 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.390936 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391004 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391024 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391041 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391141 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391192 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l78\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391241 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.391293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.391267 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492209 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492238 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492263 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492298 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492347 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l78\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492391 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492417 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492525 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.492821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.492719 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.493158 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.493133 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.493656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.493592 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.493775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.493654 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.493775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.493735 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.495151 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.495125 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.495339 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.495317 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.509388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.509357 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.509585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.509565 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l78\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.572255 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.572213 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:09.915289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.915255 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:13:09.917631 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:13:09.917578 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52641dc2_7517_4aff_aa1f_760c3f98f00e.slice/crio-5ebe6db76c2071ff95dcd3648210723511407a897a82846584ef0cac5313823f WatchSource:0}: Error finding container 5ebe6db76c2071ff95dcd3648210723511407a897a82846584ef0cac5313823f: Status 404 returned error can't find the container with id 5ebe6db76c2071ff95dcd3648210723511407a897a82846584ef0cac5313823f Oct 30 18:13:09.920347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.920309 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:09.920422 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.920402 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:09.920487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:09.920448 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:10.486647 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:13:10.486399 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:13:10.710394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:10.710362 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" event={"ID":"52641dc2-7517-4aff-aa1f-760c3f98f00e","Type":"ContainerStarted","Data":"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771"} Oct 30 18:13:10.710589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:10.710415 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" event={"ID":"52641dc2-7517-4aff-aa1f-760c3f98f00e","Type":"ContainerStarted","Data":"5ebe6db76c2071ff95dcd3648210723511407a897a82846584ef0cac5313823f"} Oct 30 18:13:11.572628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:11.572574 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:11.574356 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:11.574322 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:11.574510 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:11.574399 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:12.573087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:12.573047 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.573497 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:12.573117 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:13.573174 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:13.573128 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:13.573647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:13.573205 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:14.573327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:14.573286 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:14.573884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:14.573358 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:15.573113 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:15.573078 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:15.573306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:15.573140 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:16.573180 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:16.573141 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:13:16.573608 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:16.573199 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 18:13:17.578436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:17.578404 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:17.578951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:17.578719 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:17.579820 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:17.579797 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:13:17.626838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:17.626780 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podStartSLOduration=8.626756254 podStartE2EDuration="8.626756254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:10.768712618 +0000 UTC m=+741.589424408" watchObservedRunningTime="2026-10-30 18:13:17.626756254 +0000 UTC m=+748.447468029" Oct 30 18:13:38.812654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.807251 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:13:38.829349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.829316 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:13:38.829756 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.829683 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.841414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.841379 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.841747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.841717 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.841950 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.841932 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842170 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842149 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842358 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842524 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842739 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842806 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.842855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.842832 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxp7d\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.943886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.943839 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.943913 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.943947 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxp7d\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944088 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944119 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944149 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944181 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944219 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944258 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944280 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944678 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944786 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944760 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944780 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.944964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.944935 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.984645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.956251 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.984645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.976658 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:38.984645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:38.977855 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:39.017802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.017584 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxp7d\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:39.224415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.224373 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:39.555979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.555950 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:13:39.558996 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:13:39.558963 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de5521e_2902_4f61_9eda_b7d585099b37.slice/crio-7ec50beb56442687283cd81f82535bf8e1145764e52775cced563061e95fd954 WatchSource:0}: Error finding container 7ec50beb56442687283cd81f82535bf8e1145764e52775cced563061e95fd954: Status 404 returned error can't find the container with id 7ec50beb56442687283cd81f82535bf8e1145764e52775cced563061e95fd954 Oct 30 18:13:39.561528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.561497 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:39.561643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.561565 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:39.561643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.561616 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:39.996919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.996883 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" event={"ID":"7de5521e-2902-4f61-9eda-b7d585099b37","Type":"ContainerStarted","Data":"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26"} Oct 30 18:13:39.997433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:39.996947 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" event={"ID":"7de5521e-2902-4f61-9eda-b7d585099b37","Type":"ContainerStarted","Data":"7ec50beb56442687283cd81f82535bf8e1145764e52775cced563061e95fd954"} Oct 30 18:13:40.224939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:40.224897 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:40.226938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:40.226911 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:40.227083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:40.226963 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:40.482790 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:13:40.482745 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:13:41.225506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:41.225470 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:41.225948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:41.225525 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:42.224919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:42.224879 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:42.225135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:42.224938 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:43.225699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:43.225659 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:43.226204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:43.225724 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:44.225140 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:44.225103 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:44.225335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:44.225159 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:45.225280 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:45.225240 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:45.225717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:45.225324 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:46.224872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:46.224834 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:13:46.225085 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:46.224920 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 18:13:47.230649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:47.230613 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:47.231257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:47.231229 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:47.232138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:47.232114 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:13:47.288837 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:47.288782 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podStartSLOduration=9.288764793 podStartE2EDuration="9.288764793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:40.053905339 +0000 UTC m=+770.874617115" watchObservedRunningTime="2026-10-30 18:13:47.288764793 +0000 UTC m=+778.109476569" Oct 30 18:13:49.401179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:13:49.401137 5024 scope.go:122] "RemoveContainer" containerID="4038a90f9932adcd5a85415626c00303a32ba010cf43bd41ba899211278a2df6" Oct 30 18:14:07.884922 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:07.884878 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:14:07.889107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:07.889051 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:07.911722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:07.911683 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:14:07.917455 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:07.917412 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"distinct-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:14:08.033748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.033707 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.033945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.033821 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.033945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.033934 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttfc\" (UniqueName: \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.034047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.033982 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.034175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.034156 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.034244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.034230 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.135305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135191 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135733 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135806 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135839 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135896 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttfc\" (UniqueName: \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135936 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.136289 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.135649 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.136960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.136537 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.160625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.157534 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.162860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.162826 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttfc\" (UniqueName: \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.167736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.167703 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\") pod \"distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.215685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.215639 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:08.536787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.536752 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:14:08.539799 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:08.539769 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51412cb1_9c20_42ac_8947_642002adc88d.slice/crio-f9cbd9cc90d3a650b2e76ed2707e18f0f2f0f7dd32b80ecfbea34baa63d9ea48 WatchSource:0}: Error finding container f9cbd9cc90d3a650b2e76ed2707e18f0f2f0f7dd32b80ecfbea34baa63d9ea48: Status 404 returned error can't find the container with id f9cbd9cc90d3a650b2e76ed2707e18f0f2f0f7dd32b80ecfbea34baa63d9ea48 Oct 30 18:14:08.612479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.612449 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:14:08.616644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.616619 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.646573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.646538 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:14:08.742885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.742844 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.742885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.742887 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88w5\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.742926 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743041 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743085 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743208 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743237 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743271 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.743391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.743377 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.844923 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845025 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845093 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88w5\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845146 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845169 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845206 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845274 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845410 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845460 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845770 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.845845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.845811 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.846257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.846236 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.847641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.847586 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.847933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.847914 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.862380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.862348 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.862513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.862441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88w5\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:08.941475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:08.941438 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:09.203804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.203765 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" event={"ID":"51412cb1-9c20-42ac-8947-642002adc88d","Type":"ContainerStarted","Data":"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d"} Oct 30 18:14:09.203993 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.203823 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" event={"ID":"51412cb1-9c20-42ac-8947-642002adc88d","Type":"ContainerStarted","Data":"f9cbd9cc90d3a650b2e76ed2707e18f0f2f0f7dd32b80ecfbea34baa63d9ea48"} Oct 30 18:14:09.216168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.216136 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:09.249634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.249558 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podStartSLOduration=2.249543037 podStartE2EDuration="2.249543037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:09.247523795 +0000 UTC m=+800.068235572" watchObservedRunningTime="2026-10-30 18:14:09.249543037 +0000 UTC m=+800.070254825" Oct 30 18:14:09.274892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.274862 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:14:09.277266 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:09.277238 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bc465e_81c5_4ee1_9d36_fdd60980f9e6.slice/crio-da82aab9752dc41cfb6130f5561c62e452aaae77a4486d95cc80ca6f6674ff81 WatchSource:0}: Error finding container da82aab9752dc41cfb6130f5561c62e452aaae77a4486d95cc80ca6f6674ff81: Status 404 returned error can't find the container with id da82aab9752dc41cfb6130f5561c62e452aaae77a4486d95cc80ca6f6674ff81 Oct 30 18:14:09.280060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.280024 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:09.280161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.280102 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:09.280161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:09.280146 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:10.212480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.212430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" event={"ID":"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6","Type":"ContainerStarted","Data":"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928"} Oct 30 18:14:10.213014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.212499 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" event={"ID":"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6","Type":"ContainerStarted","Data":"da82aab9752dc41cfb6130f5561c62e452aaae77a4486d95cc80ca6f6674ff81"} Oct 30 18:14:10.275509 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.275439 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podStartSLOduration=2.275421295 podStartE2EDuration="2.275421295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:10.264086029 +0000 UTC m=+801.084797806" watchObservedRunningTime="2026-10-30 18:14:10.275421295 +0000 UTC m=+801.096133073" Oct 30 18:14:10.483324 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:14:10.483233 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:14:10.942480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.942449 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:10.943980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.943955 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:10.944103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:10.944006 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:11.942743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:11.942709 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:11.942743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:11.942762 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:12.942168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:12.942131 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:12.942367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:12.942208 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:13.942745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:13.942705 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:13.943284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:13.942789 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:14.573335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.573273 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:14:14.579954 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.579915 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.592765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.591186 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:14:14.596827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.596797 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" Oct 30 18:14:14.597154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.597124 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" Oct 30 18:14:14.604978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.604947 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:14:14.605196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.605103 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" Oct 30 18:14:14.625952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.625904 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.626141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.626027 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.626141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.626075 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.626141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.626123 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.626248 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.626229 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6v6v\" (UniqueName: \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.626335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.626320 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.727746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.727705 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.727951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.727762 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.727951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.727801 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.727951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.727853 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.727951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.727915 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6v6v\" (UniqueName: \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.728212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.728002 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.728378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.728353 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.728508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.728400 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.728584 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.728414 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.730581 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.730559 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.732935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.732895 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.745729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.745694 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6v6v\" (UniqueName: \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\") pod \"distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.902198 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.901862 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:14.943009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.942961 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:14.943687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:14.943033 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:15.244764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:15.244741 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:14:15.272292 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:15.272254 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ecb2f63_a597_4615_b3c8_60add2150149.slice/crio-abcd2201d13a4f28a5d02a479e4356355524e532e55ebeed4226caeb23bbb138 WatchSource:0}: Error finding container abcd2201d13a4f28a5d02a479e4356355524e532e55ebeed4226caeb23bbb138: Status 404 returned error can't find the container with id abcd2201d13a4f28a5d02a479e4356355524e532e55ebeed4226caeb23bbb138 Oct 30 18:14:15.290138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:15.290097 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" event={"ID":"8ecb2f63-a597-4615-b3c8-60add2150149","Type":"ContainerStarted","Data":"abcd2201d13a4f28a5d02a479e4356355524e532e55ebeed4226caeb23bbb138"} Oct 30 18:14:15.942193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:15.942133 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:15.942390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:15.942213 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 18:14:16.301831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.301015 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" event={"ID":"8ecb2f63-a597-4615-b3c8-60add2150149","Type":"ContainerStarted","Data":"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8"} Oct 30 18:14:16.337055 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.336994 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podStartSLOduration=2.336975591 podStartE2EDuration="2.336975591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:16.334639755 +0000 UTC m=+807.155351529" watchObservedRunningTime="2026-10-30 18:14:16.336975591 +0000 UTC m=+807.157687366" Oct 30 18:14:16.903133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.903092 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:16.946750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.946716 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:16.947314 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.947281 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:16.948248 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:16.948228 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:14:18.216424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:18.216386 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:18.230205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:18.230176 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:18.325270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:18.325240 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:23.782256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.782212 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:14:23.787117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.787089 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.803470 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.803441 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:14:23.808088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.808052 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.812853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.812818 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:14:23.830184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.830146 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:14:23.842241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.842193 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" Oct 30 18:14:23.868427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868384 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868669 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868459 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868669 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868498 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868667 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868755 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868795 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868828 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.868962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.868877 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.869081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.869054 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hpx5\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970304 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970397 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970443 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.970635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970569 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.970808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970692 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqggw\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.970864 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970808 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970864 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970817 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.970961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970896 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hpx5\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.970961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970921 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.970961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970943 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.970971 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.971111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971015 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971052 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971128 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971163 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.971363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971252 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971399 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971432 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971483 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.971578 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971542 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:23.971690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971668 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.971746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.971704 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.973444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.973423 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.973794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.973774 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.988851 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.988821 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:23.989026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:23.988883 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hpx5\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:24.072921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.072828 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.072921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.072889 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.072935 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.072978 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073001 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073091 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073137 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqggw\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073236 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073480 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073635 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073740 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.073831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.073814 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.074295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.074272 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.075749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.075728 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.076051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.076032 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.090040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.090008 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.090182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.090169 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqggw\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.103989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.103959 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:24.140787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.140740 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:24.426821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.426785 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:14:24.430897 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:24.430854 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba66fe8_ed81_4310_b406_9a643f2b2ba6.slice/crio-e6874e633c0e42a063c8e1ced1d9cc5900255413604d0554785aa157a72cc281 WatchSource:0}: Error finding container e6874e633c0e42a063c8e1ced1d9cc5900255413604d0554785aa157a72cc281: Status 404 returned error can't find the container with id e6874e633c0e42a063c8e1ced1d9cc5900255413604d0554785aa157a72cc281 Oct 30 18:14:24.434209 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.434094 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.434344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.434262 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.434344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.434309 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.470502 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.470471 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:14:24.473639 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:24.473591 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f575041_4a4e_42d0_a45f_c6f2f8a5fb3b.slice/crio-24792b7f5a837be0c628c46d716914b43d1bb24e906afdbf7ba2758c25644e1a WatchSource:0}: Error finding container 24792b7f5a837be0c628c46d716914b43d1bb24e906afdbf7ba2758c25644e1a: Status 404 returned error can't find the container with id 24792b7f5a837be0c628c46d716914b43d1bb24e906afdbf7ba2758c25644e1a Oct 30 18:14:24.477041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.476998 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.477187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.477095 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.477187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.477142 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:24.902962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.902929 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:24.917062 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:24.917028 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:25.400323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.400191 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" event={"ID":"7ba66fe8-ed81-4310-b406-9a643f2b2ba6","Type":"ContainerStarted","Data":"1ffb98bf9a0c686e9177d54a15ccc8a82aa7b95f42ded97436e60fc3834bafb3"} Oct 30 18:14:25.400323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.400270 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" event={"ID":"7ba66fe8-ed81-4310-b406-9a643f2b2ba6","Type":"ContainerStarted","Data":"e6874e633c0e42a063c8e1ced1d9cc5900255413604d0554785aa157a72cc281"} Oct 30 18:14:25.403038 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.402993 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" event={"ID":"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b","Type":"ContainerStarted","Data":"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d"} Oct 30 18:14:25.403175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.403063 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" event={"ID":"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b","Type":"ContainerStarted","Data":"24792b7f5a837be0c628c46d716914b43d1bb24e906afdbf7ba2758c25644e1a"} Oct 30 18:14:25.416014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.415984 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:25.437988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.437913 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podStartSLOduration=2.4378957740000002 podStartE2EDuration="2.437895774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:25.435944851 +0000 UTC m=+816.256656619" watchObservedRunningTime="2026-10-30 18:14:25.437895774 +0000 UTC m=+816.258607551" Oct 30 18:14:25.529152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:25.529081 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podStartSLOduration=2.529060107 podStartE2EDuration="2.529060107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:25.525494708 +0000 UTC m=+816.346206484" watchObservedRunningTime="2026-10-30 18:14:25.529060107 +0000 UTC m=+816.349771883" Oct 30 18:14:26.104859 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.104756 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:26.106313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.106291 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.106471 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.106353 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:26.141840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.141791 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:26.143536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.143507 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.143747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:26.143726 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:27.105186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:27.105145 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.105669 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:27.105199 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:27.142256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:27.142136 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.142256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:27.142201 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:28.106620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:28.105971 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.106620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:28.106034 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:28.141408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:28.141367 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.141591 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:28.141443 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:29.105093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:29.105053 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.105437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:29.105406 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:29.144339 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:29.144302 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.144960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:29.144924 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:30.115092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.115050 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.115394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.115362 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:30.147467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.147431 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.148176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.148099 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:30.268117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.268062 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:14:30.310766 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.309699 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.337774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.337710 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.430624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.429653 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.738823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.738707 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.843780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.843716 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.897380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:30.897338 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.005296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.003706 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:14:31.005296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.004352 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:14:31.111542 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.111048 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.111542 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.111112 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:31.151804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.151421 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.190715 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.173040 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.190715 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.173110 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:31.294694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.293758 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:14:31.294694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.294659 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:14:31.397373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.396680 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:14:31.401637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.398292 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:31.432780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.432725 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.435830 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.435159 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:14:31.436116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.436094 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:31.450930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.450898 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:31.769821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.769788 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:14:31.835049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:31.835015 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:14:32.007959 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.007907 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.017220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.017188 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:14:32.067319 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.067228 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:14:32.118380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.118254 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.122854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.122804 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:32.159473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.154323 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.159473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.154386 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:32.217664 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.200548 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:14:32.422796 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.402920 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.440188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:32.440138 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:14:33.104678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:33.104635 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:14:33.104834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:33.104710 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 18:14:33.141780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:33.141736 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:14:33.141986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:33.141794 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 18:14:34.148960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.148921 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:34.150658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.150464 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:34.152099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.152048 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:14:34.855691 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.855660 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:34.856087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.856065 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:34.856884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:34.856865 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:14:40.465220 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:14:40.465173 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:14:42.039392 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.039349 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:14:42.043970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.043944 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.056310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.056279 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:14:42.062212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.062175 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-2-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:14:42.185988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.185948 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.186199 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.186025 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.186199 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.186140 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.186329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.186202 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.186386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.186341 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.186386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.186373 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f5tz\" (UniqueName: \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300760 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f5tz\" (UniqueName: \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300840 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300922 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300956 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.301430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.300990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.306617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.304191 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.306617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.304526 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.306617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.304837 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.310617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.308320 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.328911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.327379 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f5tz\" (UniqueName: \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.335709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.335674 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\") pod \"distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.367462 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.367424 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:42.682857 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.682797 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:14:42.685743 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:42.685713 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb42b2b39_1e15_41c7_82ae_d902239de481.slice/crio-e273bc69b7f31c83870f8780c8f2bbd238329f848b74e2139a029e2525a7554d WatchSource:0}: Error finding container e273bc69b7f31c83870f8780c8f2bbd238329f848b74e2139a029e2525a7554d: Status 404 returned error can't find the container with id e273bc69b7f31c83870f8780c8f2bbd238329f848b74e2139a029e2525a7554d Oct 30 18:14:42.770416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.770372 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" event={"ID":"b42b2b39-1e15-41c7-82ae-d902239de481","Type":"ContainerStarted","Data":"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1"} Oct 30 18:14:42.770416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.770430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" event={"ID":"b42b2b39-1e15-41c7-82ae-d902239de481","Type":"ContainerStarted","Data":"e273bc69b7f31c83870f8780c8f2bbd238329f848b74e2139a029e2525a7554d"} Oct 30 18:14:42.806200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:42.806142 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podStartSLOduration=0.806124155 podStartE2EDuration="806.124155ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:42.80520393 +0000 UTC m=+833.625915708" watchObservedRunningTime="2026-10-30 18:14:42.806124155 +0000 UTC m=+833.626835931" Oct 30 18:14:43.367976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:43.367925 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:49.199637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.199559 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:14:49.204108 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.204082 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.223351 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.223315 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:14:49.229319 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.228615 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:14:49.232415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.232388 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:14:49.232415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.232394 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:14:49.232672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.232637 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"distinct-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:14:49.284026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.283989 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.284218 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.284045 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.284340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.284319 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.284392 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.284367 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf5ff\" (UniqueName: \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.284465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.284450 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.284508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.284492 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385521 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385657 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385743 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385808 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf5ff\" (UniqueName: \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385876 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.385995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.385918 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.386390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.386367 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.387346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.387005 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.387479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.387411 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.391544 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.391516 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.399026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.398997 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"distinct-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:14:49.415657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.414944 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:14:49.416572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.416540 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.442618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.433830 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:14:49.442618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.441773 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf5ff\" (UniqueName: \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\") pod \"distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.541065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.540972 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:14:49.549121 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.549090 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:49.856026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:49.855892 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:14:49.858666 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:49.858636 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0624902a_6c5d_41d0_a418_92a8b3b3baea.slice/crio-3d813d293a3c420b6b53a9dfc3149fcc769db58e263fe20b671bf118f5a570a2 WatchSource:0}: Error finding container 3d813d293a3c420b6b53a9dfc3149fcc769db58e263fe20b671bf118f5a570a2: Status 404 returned error can't find the container with id 3d813d293a3c420b6b53a9dfc3149fcc769db58e263fe20b671bf118f5a570a2 Oct 30 18:14:50.829202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:50.829167 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" event={"ID":"0624902a-6c5d-41d0-a418-92a8b3b3baea","Type":"ContainerStarted","Data":"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c"} Oct 30 18:14:50.829627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:50.829218 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" event={"ID":"0624902a-6c5d-41d0-a418-92a8b3b3baea","Type":"ContainerStarted","Data":"3d813d293a3c420b6b53a9dfc3149fcc769db58e263fe20b671bf118f5a570a2"} Oct 30 18:14:50.864443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:50.864385 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podStartSLOduration=1.864370648 podStartE2EDuration="1.864370648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:50.863042954 +0000 UTC m=+841.683754732" watchObservedRunningTime="2026-10-30 18:14:50.864370648 +0000 UTC m=+841.685082424" Oct 30 18:14:51.549689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:51.549657 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:52.368034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:52.367997 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:52.381655 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:52.381591 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:52.853750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:52.853715 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:14:53.823130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.823095 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:14:53.827753 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.827728 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.855866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.855831 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" Oct 30 18:14:53.870260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.870228 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:14:53.966645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.966565 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.966829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.966767 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.966829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.966814 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967011 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.966989 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.967091 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.967119 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7q4k\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967317 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.967288 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.967342 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:53.967465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:53.967365 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068514 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068476 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068514 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068521 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068551 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068714 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068801 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068825 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q4k\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.068918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068888 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068921 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.068947 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.069318 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069510 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.069474 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069715 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.069689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.069961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.069937 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.070169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.070151 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.071580 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.071557 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.071808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.071788 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.085709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.085639 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.085843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.085820 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q4k\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.184103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.184068 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:54.520889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.520855 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:14:54.523855 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:14:54.523823 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3996e964_79db_4b61_b069_96e161ad78b9.slice/crio-0b7dbad7e75f57bd188ee8809ba2b0ac4a82400def420eb6a50d328100e5a936 WatchSource:0}: Error finding container 0b7dbad7e75f57bd188ee8809ba2b0ac4a82400def420eb6a50d328100e5a936: Status 404 returned error can't find the container with id 0b7dbad7e75f57bd188ee8809ba2b0ac4a82400def420eb6a50d328100e5a936 Oct 30 18:14:54.526494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.526457 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:54.526634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.526538 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:54.526634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.526582 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:54.858430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.858337 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" event={"ID":"3996e964-79db-4b61-b069-96e161ad78b9","Type":"ContainerStarted","Data":"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5"} Oct 30 18:14:54.858430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.858406 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" event={"ID":"3996e964-79db-4b61-b069-96e161ad78b9","Type":"ContainerStarted","Data":"0b7dbad7e75f57bd188ee8809ba2b0ac4a82400def420eb6a50d328100e5a936"} Oct 30 18:14:54.915730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:54.915675 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podStartSLOduration=1.915660485 podStartE2EDuration="1.915660485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:54.914266136 +0000 UTC m=+845.734977912" watchObservedRunningTime="2026-10-30 18:14:54.915660485 +0000 UTC m=+845.736372260" Oct 30 18:14:55.184845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:55.184805 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:14:55.186562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:55.186536 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:14:55.186659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:55.186589 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:14:56.184862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:56.184818 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:14:56.185388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:56.184875 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:14:57.188072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:57.188033 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.188746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:57.188668 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:14:58.184497 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:58.184460 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.184733 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:58.184513 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:14:59.184735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:59.184696 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.185114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:59.184755 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:14:59.552350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:59.552253 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:59.585075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:59.585043 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:14:59.914759 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:14:59.914733 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:15:00.196700 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:00.194084 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.196700 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:00.194166 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:15:01.185015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:01.184977 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.185341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:01.185301 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:15:01.839751 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:01.839698 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.186105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.186068 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.186357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.186333 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 18:15:02.451699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.451573 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:02.774854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.774730 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.844030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.841694 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.844030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.842497 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:15:02.849433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:02.848724 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.019822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:03.019772 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:03.029232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:03.028108 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:15:03.229580 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:03.229536 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:15:03.230490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:03.230464 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:15:03.231817 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:03.231795 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:15:09.894446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.894363 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:15:09.899673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.899645 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:09.927819 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.927777 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:15:09.932815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.932663 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" Oct 30 18:15:09.932815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.932747 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" Oct 30 18:15:09.932815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.932769 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" Oct 30 18:15:09.932815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:09.932790 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"distinct-2-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:15:10.000789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.000747 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.000966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.000813 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvcs\" (UniqueName: \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.000966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.000882 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.000966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.000924 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.000966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.000949 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.001136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.001120 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102327 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102396 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102516 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102660 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvcs\" (UniqueName: \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.102878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.102747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.103289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.103255 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.103552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.103529 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.104006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.103982 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.110080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.109748 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.125407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.125342 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvcs\" (UniqueName: \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.127076 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.126908 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\") pod \"distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.240737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.240640 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:10.455271 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:15:10.455233 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:15:10.589780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:10.567620 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:15:10.589780 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:15:10.570372 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda269645_fa06_403f_a7e4_19169b6944a3.slice/crio-6855cb265d6a3e5890fadea84312f95b13beca5a7b80d3adacf9632d9866b089 WatchSource:0}: Error finding container 6855cb265d6a3e5890fadea84312f95b13beca5a7b80d3adacf9632d9866b089: Status 404 returned error can't find the container with id 6855cb265d6a3e5890fadea84312f95b13beca5a7b80d3adacf9632d9866b089 Oct 30 18:15:11.163310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:11.163276 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" event={"ID":"da269645-fa06-403f-a7e4-19169b6944a3","Type":"ContainerStarted","Data":"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc"} Oct 30 18:15:11.163310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:11.163317 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" event={"ID":"da269645-fa06-403f-a7e4-19169b6944a3","Type":"ContainerStarted","Data":"6855cb265d6a3e5890fadea84312f95b13beca5a7b80d3adacf9632d9866b089"} Oct 30 18:15:11.197841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:11.197779 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podStartSLOduration=2.197764227 podStartE2EDuration="2.197764227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:11.196407728 +0000 UTC m=+862.017119502" watchObservedRunningTime="2026-10-30 18:15:11.197764227 +0000 UTC m=+862.018476002" Oct 30 18:15:11.242004 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:11.241969 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:20.241554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:20.241520 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:20.255063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:20.255033 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:21.242922 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:21.242889 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:15:23.777138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.777098 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:15:23.783487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.783461 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.787349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.787318 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:15:23.792047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.792024 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.801626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.801587 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" Oct 30 18:15:23.806197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.806167 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:15:23.815768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.815737 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:15:23.910969 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.910919 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.910969 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.910965 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911065 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911111 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911139 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911236 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911278 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911380 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911408 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911425 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7p9\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911547 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911619 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911701 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911663 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911724 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911789 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911814 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt58p\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:23.911836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911830 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:23.911897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:23.911859 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013270 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013326 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt58p\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.013513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013391 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013485 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.013513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013511 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.013805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013545 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013579 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013627 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.013805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013656 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013823 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.013963 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014004 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014030 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014038 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014129 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014157 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7p9\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014236 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014270 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014293 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014302 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014312 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014377 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.014577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014421 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.014897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.014643 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.015044 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.015025 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.015213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.015193 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.016996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.016966 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.017182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.017163 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.017265 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.017232 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.017318 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.017261 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.031876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.031812 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.032051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.032032 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt58p\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.032295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.032269 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7p9\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.033896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.033875 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.098785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.098752 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:24.109478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.109443 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:24.452070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.452016 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.452211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.452110 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.452211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.452151 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.462145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.462113 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:15:24.466476 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.466422 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:15:24.468130 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:15:24.468100 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a4e8e2_11bc_4e2e_b1e0_b833643f7208.slice/crio-10f3971a9af66d896bded0b40d1aa8de8ab319153c358e1a4b3cec1899814a3b WatchSource:0}: Error finding container 10f3971a9af66d896bded0b40d1aa8de8ab319153c358e1a4b3cec1899814a3b: Status 404 returned error can't find the container with id 10f3971a9af66d896bded0b40d1aa8de8ab319153c358e1a4b3cec1899814a3b Oct 30 18:15:24.470978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.470943 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.471086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.471021 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.471086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:24.471064 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:25.273929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.273889 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" event={"ID":"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc","Type":"ContainerStarted","Data":"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e"} Oct 30 18:15:25.274506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.274474 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" event={"ID":"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc","Type":"ContainerStarted","Data":"23f6db9acfa129c962af89034f8538b4cd078890b5d21acf73070c5c41b70e13"} Oct 30 18:15:25.285623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.281568 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" event={"ID":"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208","Type":"ContainerStarted","Data":"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd"} Oct 30 18:15:25.285623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.281665 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" event={"ID":"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208","Type":"ContainerStarted","Data":"10f3971a9af66d896bded0b40d1aa8de8ab319153c358e1a4b3cec1899814a3b"} Oct 30 18:15:25.329087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.329024 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podStartSLOduration=2.329009135 podStartE2EDuration="2.329009135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:25.325584884 +0000 UTC m=+876.146296659" watchObservedRunningTime="2026-10-30 18:15:25.329009135 +0000 UTC m=+876.149720957" Oct 30 18:15:25.372623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:25.367751 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podStartSLOduration=2.36773065 podStartE2EDuration="2.36773065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:25.365283133 +0000 UTC m=+876.185994909" watchObservedRunningTime="2026-10-30 18:15:25.36773065 +0000 UTC m=+876.188442427" Oct 30 18:15:26.110719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.110678 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:26.112780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.112756 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.112974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.112955 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:26.113735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.113665 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:26.116545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.114717 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.116545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:26.114763 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:27.103417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:27.103166 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.103417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:27.103246 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:27.109861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:27.109831 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.110032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:27.109887 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:28.112711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:28.106017 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.112711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:28.106092 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:28.134626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:28.119132 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.134626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:28.119197 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:29.109622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:29.108889 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.109622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:29.108950 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:29.117042 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:29.110573 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.117042 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:29.110642 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:30.104258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:30.103368 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.104258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:30.103427 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:30.110709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:30.110512 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.110709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:30.110566 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:31.099801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:31.099764 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.100221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:31.099824 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 18:15:31.110329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:31.110295 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.110465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:31.110374 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 18:15:32.104563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.104532 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:32.105064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.105044 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:32.105844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.105824 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:15:32.147954 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.147924 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:32.148411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.148385 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:32.150218 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:32.150195 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:15:38.773631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.773579 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:15:38.778213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.778190 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.805180 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.805147 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:15:38.822744 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.822708 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.822965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.822756 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.822965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.822943 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823024 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823063 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823194 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823220 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823366 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823287 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.823366 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.823331 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxxj\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.924986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.924944 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.924986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.924984 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925052 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925100 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxxj\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925326 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925247 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925326 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925298 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925405 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925481 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925528 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.925667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.925646 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.926361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.926332 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.926545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.926454 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.926705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.926481 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.928033 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.928001 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.929247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.929226 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.934259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.929472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.947754 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.947725 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxxj\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:38.947941 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:38.947925 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:39.094570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:39.094470 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:39.416126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:39.416085 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:15:39.437335 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:15:39.437302 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552191e9_9850_42fc_8fc3_1bfa5c05bca5.slice/crio-759eb0e871e55c1192c82cbab58ef607c0704dd85ce4a8b4bebfe3be5799c8df WatchSource:0}: Error finding container 759eb0e871e55c1192c82cbab58ef607c0704dd85ce4a8b4bebfe3be5799c8df: Status 404 returned error can't find the container with id 759eb0e871e55c1192c82cbab58ef607c0704dd85ce4a8b4bebfe3be5799c8df Oct 30 18:15:39.440017 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:39.439974 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:39.440130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:39.440060 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:39.440130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:39.440097 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:40.462626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:40.452930 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" event={"ID":"552191e9-9850-42fc-8fc3-1bfa5c05bca5","Type":"ContainerStarted","Data":"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f"} Oct 30 18:15:40.462626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:40.453006 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" event={"ID":"552191e9-9850-42fc-8fc3-1bfa5c05bca5","Type":"ContainerStarted","Data":"759eb0e871e55c1192c82cbab58ef607c0704dd85ce4a8b4bebfe3be5799c8df"} Oct 30 18:15:40.538665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:40.537881 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podStartSLOduration=2.537860049 podStartE2EDuration="2.537860049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:40.527939072 +0000 UTC m=+891.348650849" watchObservedRunningTime="2026-10-30 18:15:40.537860049 +0000 UTC m=+891.358571826" Oct 30 18:15:40.591623 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:15:40.591471 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:15:41.111685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.105005 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:41.111866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.111738 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:41.111866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.111804 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:41.495379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.492767 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.773824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.773708 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.858764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.858709 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.925760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.925706 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.978111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:41.975695 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.034066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.033383 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:15:42.074915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.073706 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.123624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.122995 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:42.123624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.123061 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:42.151530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.150697 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.497639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.494710 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.497639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.495659 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:15:42.589653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:42.583843 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:15:43.102103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:43.101686 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:43.102103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:43.101765 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:44.095306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:44.095269 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:44.095876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:44.095321 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:45.095639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:45.095578 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:45.096049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:45.095664 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:46.095586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:46.095548 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:46.095586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:46.095620 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:47.095818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:47.095774 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:15:47.096295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:47.095864 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 18:15:48.146535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:48.146484 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:48.147258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:48.147218 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:48.148397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:48.148372 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:15:49.439112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:49.439086 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:15:49.440077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:15:49.440055 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:16:08.880946 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:08.880904 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:16:08.905665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:08.901980 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:08.960036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:08.960000 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:16:09.094624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.063534 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.063590 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.099708 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8bj\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.099815 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.099844 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.099870 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.100082 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.100157 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.113621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.100199 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.145944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.123018 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.204631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.178338 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:16:09.204919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.204898 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:16:09.205024 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.201656 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.201211 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.205226 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.205323 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205615 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.205577 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.205775 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.205908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.205894 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.206045 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.206030 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.206137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.206126 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.206272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.206260 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8bj\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.211876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.208403 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.212089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.209672 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.212669 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.212646 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:16:09.215447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.215415 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.221655 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.219897 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.224744 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.222490 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.226312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.225521 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.228490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.228413 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:16:09.228679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.228653 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.229793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.229767 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8bj\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.229905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.229880 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.315107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.314583 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.315426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.315390 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.315639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.315616 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.315852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.315823 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.316652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.316624 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.316815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.316796 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgwp\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.319186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.319055 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.319186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.319107 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.319480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.319453 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.420809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420724 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.420809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420758 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgwp\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.420809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420790 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.420809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420809 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.421160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420932 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.421160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.420999 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cg6\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.421160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.421333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421197 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.421333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421291 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.421333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421324 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.421798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421351 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.421951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421750 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.421951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421893 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.422075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.421981 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.422139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422046 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422291 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422271 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422318 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422320 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422356 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.422553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422396 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422653 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422707 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.422898 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.422880 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.423943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.423915 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.424100 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.424077 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.438775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.438739 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.439043 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.439022 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgwp\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.458976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.458941 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:09.479413 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.479385 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.523860 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.523906 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.523936 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.523987 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cg6\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.524021 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.524101 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.524241 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.524280 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.524353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.525354 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.525635 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.525911 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.527657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.526144 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.528308 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.528103 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.528870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.528841 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.530577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.530553 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.547857 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.544032 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cg6\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.565253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.565205 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.615259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.613400 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:09.796626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.796484 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:16:09.800166 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:16:09.800139 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8ae026_a8a6_4524_9cab_767bcbec09a3.slice/crio-9a513ceb535b099963807fef9ba592605f5ec6c89b5e10a2d51f787ea69c1896 WatchSource:0}: Error finding container 9a513ceb535b099963807fef9ba592605f5ec6c89b5e10a2d51f787ea69c1896: Status 404 returned error can't find the container with id 9a513ceb535b099963807fef9ba592605f5ec6c89b5e10a2d51f787ea69c1896 Oct 30 18:16:09.803219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.803183 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.803322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.803261 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.803322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.803296 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.811699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.811671 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:16:09.820980 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:16:09.820945 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9857f8a3_feef_4a67_8034_3a829c5239db.slice/crio-913397ed36d1e7fc1ef5ca00427307c4e44027c1a953f1db5fafe33bde0426aa WatchSource:0}: Error finding container 913397ed36d1e7fc1ef5ca00427307c4e44027c1a953f1db5fafe33bde0426aa: Status 404 returned error can't find the container with id 913397ed36d1e7fc1ef5ca00427307c4e44027c1a953f1db5fafe33bde0426aa Oct 30 18:16:09.824707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.824675 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.824829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.824749 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.824829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.824782 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.825417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.825394 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" event={"ID":"1a8ae026-a8a6-4524-9cab-767bcbec09a3","Type":"ContainerStarted","Data":"9a513ceb535b099963807fef9ba592605f5ec6c89b5e10a2d51f787ea69c1896"} Oct 30 18:16:09.954491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.954100 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:16:09.956010 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:16:09.955982 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81a4e6c_0678_4ac5_acd2_8c0bda7fc8d6.slice/crio-d4d175423f8c5823eef5fb505d8d14d89e89108f0ec65706184114fe241a7be8 WatchSource:0}: Error finding container d4d175423f8c5823eef5fb505d8d14d89e89108f0ec65706184114fe241a7be8: Status 404 returned error can't find the container with id d4d175423f8c5823eef5fb505d8d14d89e89108f0ec65706184114fe241a7be8 Oct 30 18:16:09.959295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.959259 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.959385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.959357 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.959423 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:09.959405 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:10.457702 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:16:10.457659 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:16:10.833998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.833891 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" event={"ID":"1a8ae026-a8a6-4524-9cab-767bcbec09a3","Type":"ContainerStarted","Data":"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0"} Oct 30 18:16:10.835875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.835840 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" event={"ID":"9857f8a3-feef-4a67-8034-3a829c5239db","Type":"ContainerStarted","Data":"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b"} Oct 30 18:16:10.836054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.835900 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" event={"ID":"9857f8a3-feef-4a67-8034-3a829c5239db","Type":"ContainerStarted","Data":"913397ed36d1e7fc1ef5ca00427307c4e44027c1a953f1db5fafe33bde0426aa"} Oct 30 18:16:10.837714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.837689 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" event={"ID":"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6","Type":"ContainerStarted","Data":"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6"} Oct 30 18:16:10.837810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.837732 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" event={"ID":"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6","Type":"ContainerStarted","Data":"d4d175423f8c5823eef5fb505d8d14d89e89108f0ec65706184114fe241a7be8"} Oct 30 18:16:10.895803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.895734 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podStartSLOduration=2.895712895 podStartE2EDuration="2.895712895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.881521763 +0000 UTC m=+921.702233539" watchObservedRunningTime="2026-10-30 18:16:10.895712895 +0000 UTC m=+921.716424673" Oct 30 18:16:10.923330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.923262 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podStartSLOduration=2.923241631 podStartE2EDuration="2.923241631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.917941874 +0000 UTC m=+921.738653651" watchObservedRunningTime="2026-10-30 18:16:10.923241631 +0000 UTC m=+921.743953408" Oct 30 18:16:10.981779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:10.981712 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podStartSLOduration=2.981690278 podStartE2EDuration="2.981690278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.979571205 +0000 UTC m=+921.800282981" watchObservedRunningTime="2026-10-30 18:16:10.981690278 +0000 UTC m=+921.802402054" Oct 30 18:16:11.471379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.471345 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:11.471832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.471804 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.475621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.471985 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:11.501036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.501006 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:11.518626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.511742 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.518626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.511897 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:11.629642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.628116 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:11.629642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.628834 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.629642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:11.628891 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:12.461084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.461041 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.461716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.461680 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:12.532915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.532875 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.533089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.532941 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:12.632295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.632258 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.632653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:12.632575 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:13.462778 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.462740 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.463382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.463337 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:13.489764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.489724 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.490065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.490038 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:13.624295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.624261 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.624570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:13.624545 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:14.482636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.470706 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.532734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.532677 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:14.586625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.518838 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.696079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.696039 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:14.696725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.641129 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.696891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:14.696870 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:15.467627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.462985 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.467627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.463052 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:15.489623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.488713 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.489623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.488781 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:15.631625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.631256 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.631625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:15.631318 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:16.658626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.657828 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.658626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.657897 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:16.659227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.658799 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.659227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.658843 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:16.659227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.658921 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.659227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:16.658942 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:17.460245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.460194 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.460462 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.460263 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 18:16:17.480223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.480183 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.480431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.480249 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 18:16:17.614809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.614763 5024 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.615512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:17.614863 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 18:16:18.465195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.465155 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:18.465739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.465629 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:18.466935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.466910 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:16:18.627732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.627683 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:18.629492 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.629456 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:18.634257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.634217 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:16:18.767640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.767535 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:18.768366 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.768332 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:18.771209 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:18.771165 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:16:36.232802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.232693 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.253814 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.253746 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.600522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.600417 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:36.778937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.777698 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.848739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.848689 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.903176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.903132 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.969743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:36.969697 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.036758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.036703 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.077850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.077803 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.143877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.143753 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:37.261683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.256766 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:37.261683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.257557 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:16:37.671637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.666027 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.712740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.707881 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:16:37.786697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.786651 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.820365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.820321 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:16:37.851041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.850998 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:16:37.853231 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.853203 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:16:37.952705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.944555 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:16:37.969625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.954026 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:16:37.973757 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.973718 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.993642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:37.992288 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:16:38.062492 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.062448 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.063394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.063367 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:16:38.087776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.087725 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:16:38.089813 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.089779 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:16:38.158636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.157697 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:38.158636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.158183 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:16:38.261688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.260121 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.464706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.456761 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:16:38.464706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.457528 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:16:38.673726 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.673678 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:38.968253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.968142 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 18:16:38.991025 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.969701 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.997057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:38.991527 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.122649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.118220 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.133839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.122964 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.160166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.159738 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.247009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.246906 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 18:16:39.283625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.266845 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:16:39.283625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.267443 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.463346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.463302 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.712625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:39.701869 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:40.101689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.097781 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.129707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.115955 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:40.135391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.135345 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:16:40.267949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.199366 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:16:40.329626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.320734 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.329626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.321920 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:16:40.329626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.322719 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.335799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.334833 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.496622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.495776 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:16:40.512640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.505200 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:16:40.512640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.508812 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:16:40.512640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.511373 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:16:40.513755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.513432 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.514624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.514257 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:16:40.527128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.527089 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:16:40.533021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.532993 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:16:40.533995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.533974 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.537630 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:16:40.535684 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:16:40.537630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.536032 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:16:40.537630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.536078 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:16:40.537630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.536109 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:16:40.537630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.537184 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:16:40.539331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.539308 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:16:40.550219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.550186 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:16:40.572216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.571978 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:16:40.572216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.572076 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.607846 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.607168 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkskv\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.608642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.608568 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.608843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.608819 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.608936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.608915 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.609055 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.609019 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.609120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.609089 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.609185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.609151 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.611212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.611179 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.611967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.611940 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713045 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713102 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713131 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713298 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713365 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713407 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713484 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713534 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713572 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713654 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713727 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713761 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713796 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713815 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713899 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.713942 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714050 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714227 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714263 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714310 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714313 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714444 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsk2\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.714489 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkskv\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.716165 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.717651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.717313 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.730564 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.730534 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.731320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.731299 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkskv\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:40.815519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815409 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.815519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815503 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsk2\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.815839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815562 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.815839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815589 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.815839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815720 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.815839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815814 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815862 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815965 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815969 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.815997 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.816060 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.816290 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.816688 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.816804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.816782 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.818425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.818403 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.818527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.818410 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.832537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.832501 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.832737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.832714 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsk2\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.871406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.871378 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:40.896934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:40.896878 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:41.202164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.202129 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:16:41.203963 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:16:41.203936 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75488f67_3857_4558_9a91_e85da8d88ea4.slice/crio-b6c3c0743b4fddcbaf1733396cb8e3770ea4e81b67a9570671ffa49521663c5a WatchSource:0}: Error finding container b6c3c0743b4fddcbaf1733396cb8e3770ea4e81b67a9570671ffa49521663c5a: Status 404 returned error can't find the container with id b6c3c0743b4fddcbaf1733396cb8e3770ea4e81b67a9570671ffa49521663c5a Oct 30 18:16:41.206708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.206580 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.206840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.206765 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.206840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.206825 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.240313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.240284 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:16:41.248645 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:16:41.248569 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88df402f_fe7b_422f_89b6_96933e761516.slice/crio-4eb7579379ec794c49b335418d9006796aa1512ddd508454ed7efdd89b74442d WatchSource:0}: Error finding container 4eb7579379ec794c49b335418d9006796aa1512ddd508454ed7efdd89b74442d: Status 404 returned error can't find the container with id 4eb7579379ec794c49b335418d9006796aa1512ddd508454ed7efdd89b74442d Oct 30 18:16:41.251543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.251511 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.251693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.251590 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.251693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.251667 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:41.581506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.581386 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" event={"ID":"88df402f-fe7b-422f-89b6-96933e761516","Type":"ContainerStarted","Data":"839421118a5d4e29abe432b8ed169b46ed94654ede5bd81e144f82606a2944e7"} Oct 30 18:16:41.582146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.582111 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" event={"ID":"88df402f-fe7b-422f-89b6-96933e761516","Type":"ContainerStarted","Data":"4eb7579379ec794c49b335418d9006796aa1512ddd508454ed7efdd89b74442d"} Oct 30 18:16:41.583688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.583657 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" event={"ID":"75488f67-3857-4558-9a91-e85da8d88ea4","Type":"ContainerStarted","Data":"de151b65290dc1a7246c8810035fafd92382432bb3d3cdabe3d3da168ffa5514"} Oct 30 18:16:41.583802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.583713 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" event={"ID":"75488f67-3857-4558-9a91-e85da8d88ea4","Type":"ContainerStarted","Data":"b6c3c0743b4fddcbaf1733396cb8e3770ea4e81b67a9570671ffa49521663c5a"} Oct 30 18:16:41.632476 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.632417 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podStartSLOduration=2.632401744 podStartE2EDuration="2.632401744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:41.627266981 +0000 UTC m=+952.447978771" watchObservedRunningTime="2026-10-30 18:16:41.632401744 +0000 UTC m=+952.453113519" Oct 30 18:16:41.665758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.665693 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podStartSLOduration=3.665671414 podStartE2EDuration="3.665671414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:41.662673552 +0000 UTC m=+952.483385330" watchObservedRunningTime="2026-10-30 18:16:41.665671414 +0000 UTC m=+952.486383193" Oct 30 18:16:41.888407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.885725 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:41.899731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:41.899701 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:42.906937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:42.906900 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:42.907568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:42.907531 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:42.908704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:42.908679 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:42.908904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:42.908877 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:43.885057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:43.885019 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:43.885344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:43.885317 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:43.897625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:43.897449 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:43.897625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:43.897535 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:44.872493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:44.872039 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.872493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:44.872104 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:44.906201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:44.906170 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.906427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:44.906408 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:45.877054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:45.877012 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.877738 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:45.877700 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:45.903626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:45.902711 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.903626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:45.902779 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:46.904815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:46.904773 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.905447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:46.905415 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:46.905555 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:46.904821 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.905843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:46.905797 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:47.872450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:47.872407 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.872689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:47.872473 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 18:16:47.897667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:47.897582 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.897890 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:47.897713 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 18:16:48.876762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.876723 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:48.877432 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.877408 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:48.878135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.878117 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:16:48.902658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.902630 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:48.902999 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.902977 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:48.903710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:48.903692 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:16:58.157624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.156704 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.190626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.189643 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.196623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.191317 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.286093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.272713 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:16:58.360630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.353781 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.439886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.439755 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.518782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.518719 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.546012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.545967 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.549660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.549591 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.697686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.693778 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:58.995943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:58.995812 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.172802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.170709 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.172802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.171576 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:16:59.210638 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.196290 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:16:59.210638 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.198885 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:16:59.224640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.222252 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.274827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.274700 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.275877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.275849 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:16:59.481803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.481760 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.495691 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.488070 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:16:59.535448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.535347 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.571804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.571754 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.573919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.573550 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.724627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.712365 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:16:59.724627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.712446 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:16:59.724627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.712491 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:16:59.724627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.712527 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:16:59.724627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.712559 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:16:59.745683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.731329 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.745683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:16:59.731871 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:00.184716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.184671 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.288732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.288683 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:17:00.458682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.432835 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.458682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.434443 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.509624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.501804 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:17:00.542777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.542721 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.570625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.545692 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.583754 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.583698 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.660930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.584729 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.661447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.584777 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 18:17:00.752158 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.752057 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.775117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.775070 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.838901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.838852 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.916621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:00.891893 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.300400 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.300355 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.320933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.301007 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.446174 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.446126 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:17:01.446706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.446676 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:17:01.467008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.466964 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.516624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.510699 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.595270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.576701 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:01.621791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.621725 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:01.660708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.660643 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:17:01.677626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.670222 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:01.677626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:01.671757 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:17:02.078734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.078667 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.287623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.281976 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:17:02.287623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.284016 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:02.337683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.325688 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:17:02.373816 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.373759 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:02.376116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.374110 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.471657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.455668 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.501653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.493705 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.516409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.516369 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.591900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.591815 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:17:02.618446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.618414 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:17:02.632137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.632108 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:17:02.632726 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.632225 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.677801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.677741 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:02.686648 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.680898 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:02.729159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.729135 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:17:02.747667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.747624 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:17:02.834488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.834457 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:02.846535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:02.846447 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:17:09.080952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.080912 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:17:09.087656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.087198 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.110699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110660 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.110872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110713 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.110872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110771 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.110872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110807 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.111060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110893 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.111060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110950 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69nd\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.111060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.110988 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.111060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.111027 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.111276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.111085 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.113974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.113944 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:17:09.212770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212737 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.212974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212776 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.212974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212818 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.212974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212852 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.212974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212932 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.212975 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69nd\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213016 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213054 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213111 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213462 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213341 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213728 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213686 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213733 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213779 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.213932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.213864 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.215250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.215229 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.215902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.215879 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.230784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.230756 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.230948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.230852 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69nd\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.415377 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.415337 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:09.741101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.741071 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:17:09.743530 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:17:09.743500 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe5dff0_720b_42a4_b3c1_98943b0fc758.slice/crio-8f5e0cde5afdb57c9b1c51844366b1d5868a5a955ee4f73057cb20661a12c4c0 WatchSource:0}: Error finding container 8f5e0cde5afdb57c9b1c51844366b1d5868a5a955ee4f73057cb20661a12c4c0: Status 404 returned error can't find the container with id 8f5e0cde5afdb57c9b1c51844366b1d5868a5a955ee4f73057cb20661a12c4c0 Oct 30 18:17:09.746099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.746066 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:09.746187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.746135 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:09.746187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:09.746166 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:10.187215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.187177 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" event={"ID":"3fe5dff0-720b-42a4-b3c1-98943b0fc758","Type":"ContainerStarted","Data":"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268"} Oct 30 18:17:10.187729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.187235 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" event={"ID":"3fe5dff0-720b-42a4-b3c1-98943b0fc758","Type":"ContainerStarted","Data":"8f5e0cde5afdb57c9b1c51844366b1d5868a5a955ee4f73057cb20661a12c4c0"} Oct 30 18:17:10.230696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.230647 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podStartSLOduration=1.230631246 podStartE2EDuration="1.230631246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:10.226505401 +0000 UTC m=+981.047217170" watchObservedRunningTime="2026-10-30 18:17:10.230631246 +0000 UTC m=+981.051343025" Oct 30 18:17:10.415647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.415555 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:10.416893 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.416852 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:10.417027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:10.416915 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:10.488348 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:17:10.488197 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:17:11.416137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:11.416101 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:11.416650 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:11.416163 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:12.416123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:12.416084 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:12.416307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:12.416144 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:13.416427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:13.416390 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:13.416880 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:13.416446 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:14.416003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:14.415970 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:14.416208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:14.416026 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:15.415961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:15.415928 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:15.416476 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:15.415985 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:16.415878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:16.415842 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:17:16.416060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:16.415908 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 18:17:17.420620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:17.420571 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:17.421085 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:17.421000 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:17.421947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:17.421927 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:17:24.133073 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.133032 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:17:24.140260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.140221 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.179023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.178976 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnrv\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.179459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.179431 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.179641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.179622 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.179861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.179842 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.180000 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.179983 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.180110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.180093 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.180215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.180201 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.180377 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.180362 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.180483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.180469 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.190664 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.190621 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:17:24.281985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.281936 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.281985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.281984 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282052 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282081 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282116 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282168 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282192 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.282274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.282242 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnrv\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.283218 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.283186 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.283707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.283660 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.284833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.283811 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.284833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.284064 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.284833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.284518 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.291233 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.291208 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.291460 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.291438 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.302314 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.302280 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.302457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.302378 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnrv\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:24.500623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:24.499228 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:25.937637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:25.927877 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 18:17:26.047625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.034312 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:17:26.165563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.165511 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:26.165895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.165868 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:26.166051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.166032 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:17:26.326681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.316651 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.348645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.344369 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.501685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.474583 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.567713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.564016 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" event={"ID":"f925cf97-755f-4bd8-adfe-3d406b99270a","Type":"ContainerStarted","Data":"be473864a29d6090dd4e56440cee0346ef633fa46cf01efedcbfd08a895fb46a"} Oct 30 18:17:26.699215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.697490 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:17:26.700940 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.700896 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.732671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.727125 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.740637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.737568 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:26.765706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.757753 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.766793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:26.766709 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.489862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.450387 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.489862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.452461 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:17:27.489862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.475765 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.535579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.507158 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.614083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.614035 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:27.614895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.614839 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:17:27.615144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.615099 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:17:27.773736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.771737 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.773736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.772500 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:17:27.774099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.774048 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.774704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.774686 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:17:27.789611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.777277 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.789611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.778688 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.789611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.778767 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:27.789611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.778863 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.790714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.790688 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:17:27.904625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.791794 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:17:27.904625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.791888 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:17:27.904625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:27.791927 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:28.310977 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.310950 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:28.395672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.395638 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" event={"ID":"f925cf97-755f-4bd8-adfe-3d406b99270a","Type":"ContainerStarted","Data":"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7"} Oct 30 18:17:28.472512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.456952 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:17:28.480796 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.480740 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.512116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.507269 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:28.512116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.507374 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.512116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.508111 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:28.555647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.544125 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:28.555647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.544185 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:28.809574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.796974 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.809574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.797344 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:17:28.809574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.808765 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:28.842363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.810012 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:28.868761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.867422 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:17:28.868761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.868500 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:17:28.991894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:28.991847 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.110680 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.066568 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podStartSLOduration=5.066547721 podStartE2EDuration="5.066547721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:28.543434329 +0000 UTC m=+999.364146105" watchObservedRunningTime="2026-10-30 18:17:29.066547721 +0000 UTC m=+999.887259500" Oct 30 18:17:29.377682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.342583 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.570707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.540965 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.652681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.574958 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.665632 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.661034 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:17:29.808625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.801792 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.839501 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.839457 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:29.839812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.839771 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:29.876666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.860792 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:29.892219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.892170 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.901333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.899691 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:17:29.976643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.898625 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:30.002585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:29.898403 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:30.250684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.226111 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 30 18:17:30.412625 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:17:30.408144 5024 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s" Oct 30 18:17:30.502612 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.502475 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:30.508761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.508699 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:30.748324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.746290 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:17:30.802851 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.802741 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.820671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.810634 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.821227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.812977 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:17:30.824070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.824040 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:30.824261 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.813015 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 18:17:30.856632 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.847560 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:30.862232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.861037 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.907499 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.907451 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.914919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.914771 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:17:30.917969 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.917933 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.944770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.944741 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:30.946568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.946546 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:30.951646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.951614 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:17:30.960522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.960120 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.984624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.983560 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.991322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:30.991294 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:17:31.006808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:31.006769 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:17:31.008581 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:31.008554 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:17:31.009685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:31.009654 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:17:31.500686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:31.500649 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:31.500870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:31.500715 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:32.500928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:32.500828 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:32.500928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:32.500887 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:33.500646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:33.500574 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:33.500848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:33.500674 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:34.500864 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:34.500826 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:34.501278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:34.500891 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:35.500915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:35.500875 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:17:35.501315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:35.500945 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 18:17:36.053060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.053024 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7277-8df2b9"] Oct 30 18:17:36.057490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.057468 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:36.070747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.070718 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7277-8df2b9"] Oct 30 18:17:36.076326 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.076296 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n7rzn\"" Oct 30 18:17:36.223353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.223306 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzpb\" (UniqueName: \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\") pod \"test-curl-7277-8df2b9\" (UID: \"68e2c6d0-0482-4368-99a3-bd8dd5c84798\") " pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:36.325173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.325060 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzpb\" (UniqueName: \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\") pod \"test-curl-7277-8df2b9\" (UID: \"68e2c6d0-0482-4368-99a3-bd8dd5c84798\") " pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:36.342106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.342069 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzpb\" (UniqueName: \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\") pod \"test-curl-7277-8df2b9\" (UID: \"68e2c6d0-0482-4368-99a3-bd8dd5c84798\") " pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:36.372552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.372519 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:36.685007 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:17:36.684964 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68e2c6d0_0482_4368_99a3_bd8dd5c84798.slice/crio-acbf3792679de75df411c002cbe4e7e34ea5aae35838716613912006a3b72139 WatchSource:0}: Error finding container acbf3792679de75df411c002cbe4e7e34ea5aae35838716613912006a3b72139: Status 404 returned error can't find the container with id acbf3792679de75df411c002cbe4e7e34ea5aae35838716613912006a3b72139 Oct 30 18:17:36.687159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.687138 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:17:36.687743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.687713 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7277-8df2b9"] Oct 30 18:17:36.788037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.788002 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:36.788668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.788638 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:36.790246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:36.790224 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:17:37.241152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:37.241109 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7277-8df2b9" event={"ID":"68e2c6d0-0482-4368-99a3-bd8dd5c84798","Type":"ContainerStarted","Data":"acbf3792679de75df411c002cbe4e7e34ea5aae35838716613912006a3b72139"} Oct 30 18:17:39.257343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:39.257292 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7277-8df2b9" event={"ID":"68e2c6d0-0482-4368-99a3-bd8dd5c84798","Type":"ContainerStarted","Data":"0d6571cccdc413a3d5eeea049940e2560722f5f653104c6d72d5fa6bc41d5dd8"} Oct 30 18:17:39.291486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:39.291412 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7277-8df2b9" podStartSLOduration=1.2940009479999999 podStartE2EDuration="3.291392367s" totalImagesPullingTime="1.997391419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:39.289808964 +0000 UTC m=+1010.110520743" watchObservedRunningTime="2026-10-30 18:17:39.291392367 +0000 UTC m=+1010.112104143" Oct 30 18:17:40.162242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.160778 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7277-8df2b9"] Oct 30 18:17:40.166222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.165115 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7277-8df2b9"] Oct 30 18:17:40.331675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.320742 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7277-7649da"] Oct 30 18:17:40.331675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.321883 5024 cpu_manager.go:414] "containerMap: removing container" podUID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" containerName="test-curl-7277-8df2b9" Oct 30 18:17:40.331675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.321905 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" containerName="test-curl-7277-8df2b9" Oct 30 18:17:40.331675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.322097 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" containerName="test-curl-7277-8df2b9" Oct 30 18:17:40.331675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.331335 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:40.434243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.434154 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7277-7649da"] Oct 30 18:17:40.517105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.517061 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlvg\" (UniqueName: \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\") pod \"test-curl-7277-7649da\" (UID: \"4124cc91-ab7e-4ce7-bdb2-34fe79f48152\") " pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:40.627113 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.627075 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlvg\" (UniqueName: \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\") pod \"test-curl-7277-7649da\" (UID: \"4124cc91-ab7e-4ce7-bdb2-34fe79f48152\") " pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:40.678993 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.678897 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlvg\" (UniqueName: \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\") pod \"test-curl-7277-7649da\" (UID: \"4124cc91-ab7e-4ce7-bdb2-34fe79f48152\") " pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:40.749446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:40.739725 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:40.995041 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:17:40.994976 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:17:41.527065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:41.499404 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.529692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:41.499887 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7277-8df2b9" podUID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" containerName="test-curl-7277-8df2b9" containerID="cri-o://0d6571cccdc413a3d5eeea049940e2560722f5f653104c6d72d5fa6bc41d5dd8" gracePeriod=2 Oct 30 18:17:41.560629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:41.560547 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.688829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:41.683718 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.959632 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:41.958723 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.034258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.034211 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.154645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.152544 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7277-7649da"] Oct 30 18:17:42.208624 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:17:42.204890 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4124cc91_ab7e_4ce7_bdb2_34fe79f48152.slice/crio-e2071b43261c18ebeb5fde2d8b2c2c70ded5874b4cd467bef80e1127af3c73ee WatchSource:0}: Error finding container e2071b43261c18ebeb5fde2d8b2c2c70ded5874b4cd467bef80e1127af3c73ee: Status 404 returned error can't find the container with id e2071b43261c18ebeb5fde2d8b2c2c70ded5874b4cd467bef80e1127af3c73ee Oct 30 18:17:42.484757 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.484706 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.569660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.567060 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.569660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.567671 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:42.593962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.591051 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7277-7649da" event={"ID":"4124cc91-ab7e-4ce7-bdb2-34fe79f48152","Type":"ContainerStarted","Data":"e2071b43261c18ebeb5fde2d8b2c2c70ded5874b4cd467bef80e1127af3c73ee"} Oct 30 18:17:42.608174 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.608145 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:17:42.718902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.703403 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.718902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.703981 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:42.803853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.803701 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.804457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.804425 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:17:42.960776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.960727 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.984625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:42.961783 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:17:43.042632 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.036450 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.066707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.062140 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:17:43.066707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.063129 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.430625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.425730 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.501589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.501537 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:17:43.843914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.843854 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:43.886695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.872845 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.039725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.877706 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:17:44.039725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:43.982393 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:44.039725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.004531 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:17:44.070634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.067064 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.070634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.067464 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:17:44.111764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.111707 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:17:44.222418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.222375 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:17:44.223092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.223064 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:17:44.224094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.224067 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tbfd4" podUID="92ce5a81-1b4a-4941-90a5-5a1e38ed60b0" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.42:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.502077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.502030 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.796463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.505429 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:17:44.796463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.526033 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:17:44.796463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.789940 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:17:44.796463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.790011 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:17:44.881152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.881102 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:17:44.883310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.883282 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:17:44.919379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.919348 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:17:44.946110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.946071 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.986024 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:44.984696 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:17:45.002812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.002772 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:17:45.012768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.012723 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.160074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.160043 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:17:45.211540 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.211504 5024 generic.go:356] "Generic (PLEG): container finished" podID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" containerID="0d6571cccdc413a3d5eeea049940e2560722f5f653104c6d72d5fa6bc41d5dd8" exitCode=137 Oct 30 18:17:45.642050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.642016 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:45.714937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.714835 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\" (UniqueName: \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\") pod \"68e2c6d0-0482-4368-99a3-bd8dd5c84798\" (UID: \"68e2c6d0-0482-4368-99a3-bd8dd5c84798\") " Oct 30 18:17:45.798511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.798465 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.906869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.905654 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.947625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.923419 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:17:45.966874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.966733 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:17:45.967216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.967201 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:17:45.968352 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:45.955039 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb" pod "68e2c6d0-0482-4368-99a3-bd8dd5c84798" (UID: "68e2c6d0-0482-4368-99a3-bd8dd5c84798"). InnerVolumeSpecName "kube-api-access-dpzpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:46.034546 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:46.034509 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzpb\" (UniqueName: \"kubernetes.io/projected/68e2c6d0-0482-4368-99a3-bd8dd5c84798-kube-api-access-dpzpb\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:17:46.220459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:46.220376 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-8df2b9" Oct 30 18:17:46.220459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:46.220430 5024 scope.go:122] "RemoveContainer" containerID="0d6571cccdc413a3d5eeea049940e2560722f5f653104c6d72d5fa6bc41d5dd8" Oct 30 18:17:46.222332 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:46.222302 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7277-7649da" event={"ID":"4124cc91-ab7e-4ce7-bdb2-34fe79f48152","Type":"ContainerStarted","Data":"bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad"} Oct 30 18:17:46.257100 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:46.257032 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7277-7649da" podStartSLOduration=6.257012858 podStartE2EDuration="6.257012858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:46.254950726 +0000 UTC m=+1017.075662501" watchObservedRunningTime="2026-10-30 18:17:46.257012858 +0000 UTC m=+1017.077724636" Oct 30 18:17:47.147928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:47.147862 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7277-7649da"] Oct 30 18:17:47.151702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:47.151665 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7277-7649da"] Oct 30 18:17:47.308179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:47.308146 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e2c6d0-0482-4368-99a3-bd8dd5c84798" path="/var/lib/kubelet/pods/68e2c6d0-0482-4368-99a3-bd8dd5c84798/volumes" Oct 30 18:17:48.237724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:48.237686 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7277-7649da" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" containerName="test-curl-7277-7649da" containerID="cri-o://bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad" gracePeriod=2 Oct 30 18:17:50.431926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:50.431901 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:50.505964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:50.505878 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\" (UniqueName: \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\") pod \"4124cc91-ab7e-4ce7-bdb2-34fe79f48152\" (UID: \"4124cc91-ab7e-4ce7-bdb2-34fe79f48152\") " Oct 30 18:17:50.508786 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:50.508747 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg" pod "4124cc91-ab7e-4ce7-bdb2-34fe79f48152" (UID: "4124cc91-ab7e-4ce7-bdb2-34fe79f48152"). InnerVolumeSpecName "kube-api-access-8vlvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:50.614444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:50.614404 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlvg\" (UniqueName: \"kubernetes.io/projected/4124cc91-ab7e-4ce7-bdb2-34fe79f48152-kube-api-access-8vlvg\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.264360 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.264326 5024 generic.go:356] "Generic (PLEG): container finished" podID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" containerID="bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad" exitCode=137 Oct 30 18:17:51.264556 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.264386 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:51.264556 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.264435 5024 scope.go:122] "RemoveContainer" containerID="bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad" Oct 30 18:17:51.275916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.275894 5024 scope.go:122] "RemoveContainer" containerID="bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad" Oct 30 18:17:51.276192 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:17:51.276173 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad\": container with ID starting with bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad not found: ID does not exist" containerID="bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad" Oct 30 18:17:51.276254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.276203 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad"} err="failed to get container status \"bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad\": rpc error: code = NotFound desc = could not find container \"bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad\": container with ID starting with bfedeb919b390b455f45f1dabca5b3a43e54fd822b192952ff0d76db986289ad not found: ID does not exist" Oct 30 18:17:51.285700 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:17:51.285629 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7277-7649da\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" pod="opendatahub/test-curl-7277-7649da" Oct 30 18:17:51.307330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:17:51.307295 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" path="/var/lib/kubelet/pods/4124cc91-ab7e-4ce7-bdb2-34fe79f48152/volumes" Oct 30 18:18:02.420083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.420004 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:18:02.420696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.420681 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" containerName="test-curl-7277-7649da" Oct 30 18:18:02.420750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.420698 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" containerName="test-curl-7277-7649da" Oct 30 18:18:02.420820 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.420811 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4124cc91-ab7e-4ce7-bdb2-34fe79f48152" containerName="test-curl-7277-7649da" Oct 30 18:18:02.426001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.425977 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.441773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.441747 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:18:02.447411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.447375 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7cb9c50e\"" Oct 30 18:18:02.447585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.447478 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7cb9c50e-dockercfg-cxczq\"" Oct 30 18:18:02.526757 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.526717 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.527188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.527162 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dnh\" (UniqueName: \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.636662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.635744 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dnh\" (UniqueName: \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.648542 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.648503 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.652032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.651517 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.676311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.676245 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dnh\" (UniqueName: \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\") pod \"payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:02.745916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:02.745883 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:03.102077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:03.102045 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:18:03.104370 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:03.104345 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb1828b_92ad_4bee_b7b1_eb041cbe3c96.slice/crio-a6afc19230d0afa133859ec4e04377059d7b0dd83dd3ea26bcca104c3d4a4de1 WatchSource:0}: Error finding container a6afc19230d0afa133859ec4e04377059d7b0dd83dd3ea26bcca104c3d4a4de1: Status 404 returned error can't find the container with id a6afc19230d0afa133859ec4e04377059d7b0dd83dd3ea26bcca104c3d4a4de1 Oct 30 18:18:03.370946 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:03.370853 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" event={"ID":"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96","Type":"ContainerStarted","Data":"a6afc19230d0afa133859ec4e04377059d7b0dd83dd3ea26bcca104c3d4a4de1"} Oct 30 18:18:05.539624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.534705 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.618624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.617703 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.922363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.908822 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:05.922363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.908908 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.922363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.910199 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.959626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.955695 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.959626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.957104 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.001627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:05.995701 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:18:06.023623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.018824 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:18:06.023623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.019212 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:18:06.134483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.117450 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.176690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.165461 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.528695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.525738 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.589626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.578732 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.589626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.578951 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:18:06.633623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.632724 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.671624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:06.670734 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.050049 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.051972 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.052291 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.054947 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.055003 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:07.098450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.055033 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:18:07.128329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.117721 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.128329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.119093 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:18:07.128329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.120686 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.128329 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.121347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:07.351586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.350295 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:18:07.382623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.353308 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:18:07.609681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.591684 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.666623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.666370 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.925438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.762585 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.925438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.923749 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:07.925438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:07.923830 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.119802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.119749 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.183484 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.183380 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 18:18:08.248624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.237977 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 18:18:08.252622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.249902 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:18:08.252622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.249969 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:08.454837 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.454725 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.489683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.458081 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:18:08.631621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.615522 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:18:08.669621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.669178 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.807399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.807279 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:18:08.856682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.812291 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.929737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:08.929692 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 18:18:09.044629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.044568 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 18:18:09.167040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.166991 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:09.194446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.194402 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.256730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.256308 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.257034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.257003 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.457791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.457688 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.551621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.543957 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:18:09.665893 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.665049 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 18:18:09.747957 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.747855 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.755750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.755720 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:18:09.836779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.836732 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:18:09.838652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.838627 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:09.905611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:09.888129 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:18:10.050279 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.050149 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.051332 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.051146 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:18:10.092036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.091801 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:18:10.093403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.093381 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:18:10.227251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.227214 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:18:10.227501 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.227484 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:18:10.231118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.231054 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:18:10.236328 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.236284 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.237210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.237185 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:10.278689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.265729 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.299293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.299261 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:18:10.306235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:10.306157 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:10.499411 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:18:10.499375 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:18:11.488848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.488816 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:18:11.500306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.500280 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.510083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.510037 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:18:11.513490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.513461 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f6a224e7\"" Oct 30 18:18:11.513678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.513660 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f6a224e7-dockercfg-r6hrv\"" Oct 30 18:18:11.577774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.577734 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cps2\" (UniqueName: \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.577944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.577809 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.581998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.581965 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:18:11.590130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.590094 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.608168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.608135 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:18:11.678940 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.678903 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.679193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.678977 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cps2\" (UniqueName: \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.679193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.679023 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.679193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.679131 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mp5\" (UniqueName: \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.680460 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.680437 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.697557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.697487 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cps2\" (UniqueName: \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\") pod \"payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.781364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.781269 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.781547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.781443 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mp5\" (UniqueName: \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.782254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.782227 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.800770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.800733 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mp5\" (UniqueName: \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\") pod \"payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:11.821442 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.821412 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:11.914495 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:11.914170 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:12.204661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.204626 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:18:12.215248 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:12.215211 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c207332_4822_42fc_a08e_420916c949d2.slice/crio-7b39e84b427f33664efb0a6c052dad9cab6faecf190004e80f8ed0e158740276 WatchSource:0}: Error finding container 7b39e84b427f33664efb0a6c052dad9cab6faecf190004e80f8ed0e158740276: Status 404 returned error can't find the container with id 7b39e84b427f33664efb0a6c052dad9cab6faecf190004e80f8ed0e158740276 Oct 30 18:18:12.251518 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.251486 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:18:12.257616 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:12.257568 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d15a2a0_f6cb_4d9b_a449_2b2c6910a78a.slice/crio-aa151d81e16b9e9091c89898c02aa2bee618ee5c0e70315a5d242d8ec4f076e4 WatchSource:0}: Error finding container aa151d81e16b9e9091c89898c02aa2bee618ee5c0e70315a5d242d8ec4f076e4: Status 404 returned error can't find the container with id aa151d81e16b9e9091c89898c02aa2bee618ee5c0e70315a5d242d8ec4f076e4 Oct 30 18:18:12.987369 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.987330 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" event={"ID":"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a","Type":"ContainerStarted","Data":"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7"} Oct 30 18:18:12.987873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.987387 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" event={"ID":"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a","Type":"ContainerStarted","Data":"aa151d81e16b9e9091c89898c02aa2bee618ee5c0e70315a5d242d8ec4f076e4"} Oct 30 18:18:12.987873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.987429 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:12.989737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.989706 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" event={"ID":"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96","Type":"ContainerStarted","Data":"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265"} Oct 30 18:18:12.989876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.989768 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:12.991307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.991285 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" event={"ID":"9c207332-4822-42fc-a08e-420916c949d2","Type":"ContainerStarted","Data":"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d"} Oct 30 18:18:12.991376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.991328 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" event={"ID":"9c207332-4822-42fc-a08e-420916c949d2","Type":"ContainerStarted","Data":"7b39e84b427f33664efb0a6c052dad9cab6faecf190004e80f8ed0e158740276"} Oct 30 18:18:12.991376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:12.991356 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:13.028899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:13.028845 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" podStartSLOduration=2.028830439 podStartE2EDuration="2.028830439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:13.024913535 +0000 UTC m=+1043.845625315" watchObservedRunningTime="2026-10-30 18:18:13.028830439 +0000 UTC m=+1043.849542215" Oct 30 18:18:13.060723 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:13.060664 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" podStartSLOduration=1.938614634 podStartE2EDuration="11.060649076s" totalImagesPullingTime="9.122034442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:13.059087148 +0000 UTC m=+1043.879798924" watchObservedRunningTime="2026-10-30 18:18:13.060649076 +0000 UTC m=+1043.881360852" Oct 30 18:18:13.096274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:13.096218 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" podStartSLOduration=1.688356245 podStartE2EDuration="2.096202233s" totalImagesPullingTime="407.845988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:13.093649423 +0000 UTC m=+1043.914361198" watchObservedRunningTime="2026-10-30 18:18:13.096202233 +0000 UTC m=+1043.916914048" Oct 30 18:18:16.814321 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.814283 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:18:16.819499 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.819476 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.836195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.836167 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-535b743b-serving-cert\"" Oct 30 18:18:16.852523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.852493 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:18:16.873679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.873591 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.874479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.873732 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.874479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.873796 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7j45\" (UniqueName: \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.874479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.874089 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.975353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.975309 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7j45\" (UniqueName: \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.975553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.975512 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.975707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.975691 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.975784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.975752 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.976250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.976230 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.978430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.978399 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.978527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.978399 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:16.991679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:16.991656 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7j45\" (UniqueName: \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\") pod \"maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:17.138574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:17.138481 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:17.470185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:17.470159 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:18:17.472843 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:17.472814 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919d62aa_b58d_403e_aadf_794515ddc80b.slice/crio-d45f2a27881eec26ac290fe602885a7d0f1a2ee5b09803795d9cde4d7b2817c6 WatchSource:0}: Error finding container d45f2a27881eec26ac290fe602885a7d0f1a2ee5b09803795d9cde4d7b2817c6: Status 404 returned error can't find the container with id d45f2a27881eec26ac290fe602885a7d0f1a2ee5b09803795d9cde4d7b2817c6 Oct 30 18:18:18.029822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:18.029504 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" event={"ID":"919d62aa-b58d-403e-aadf-794515ddc80b","Type":"ContainerStarted","Data":"d45f2a27881eec26ac290fe602885a7d0f1a2ee5b09803795d9cde4d7b2817c6"} Oct 30 18:18:19.047683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:19.047633 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" event={"ID":"919d62aa-b58d-403e-aadf-794515ddc80b","Type":"ContainerStarted","Data":"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e"} Oct 30 18:18:19.048093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:19.047743 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:19.088244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:19.088189 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" podStartSLOduration=2.705714675 podStartE2EDuration="3.088174182s" totalImagesPullingTime="382.459507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:19.083634702 +0000 UTC m=+1049.904346469" watchObservedRunningTime="2026-10-30 18:18:19.088174182 +0000 UTC m=+1049.908885957" Oct 30 18:18:23.313511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.313475 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:18:23.323133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.321846 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.328800 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.328763 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:18:23.337033 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.336982 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7cb9c50e-serving-cert\"" Oct 30 18:18:23.355889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.355852 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92w7\" (UniqueName: \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.356098 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.356002 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.356338 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.356311 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.356467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.356387 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.458648 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.458591 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.458861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.458700 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.458861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.458785 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92w7\" (UniqueName: \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.458861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.458842 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.459538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.459424 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.461629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.461582 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.461733 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.461659 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.478053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.478026 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92w7\" (UniqueName: \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\") pod \"maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.645787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.645752 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:23.957674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:23.957645 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:18:23.960233 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:23.960195 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99dea5e_f41d_4c76_8611_ae8e0b1985bd.slice/crio-56372292ed33e3c2f9f3909e003da8bc92ed3ff2dc71596c0c14a5993d8a3009 WatchSource:0}: Error finding container 56372292ed33e3c2f9f3909e003da8bc92ed3ff2dc71596c0c14a5993d8a3009: Status 404 returned error can't find the container with id 56372292ed33e3c2f9f3909e003da8bc92ed3ff2dc71596c0c14a5993d8a3009 Oct 30 18:18:24.001760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:24.001730 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:18:24.001912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:24.001821 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:18:24.001912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:24.001867 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:18:24.093045 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:24.093013 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" event={"ID":"c99dea5e-f41d-4c76-8611-ae8e0b1985bd","Type":"ContainerStarted","Data":"56372292ed33e3c2f9f3909e003da8bc92ed3ff2dc71596c0c14a5993d8a3009"} Oct 30 18:18:25.061696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:25.061669 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:18:25.104612 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:25.104552 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" event={"ID":"c99dea5e-f41d-4c76-8611-ae8e0b1985bd","Type":"ContainerStarted","Data":"5691e9d27021b03ff959f999806e2cac61e0ab02d154a9e84162a7f5858ae763"} Oct 30 18:18:25.104782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:25.104668 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:25.157330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:25.157261 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" podStartSLOduration=1.77274294 podStartE2EDuration="2.157237615s" totalImagesPullingTime="384.494675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:25.156427213 +0000 UTC m=+1055.977138990" watchObservedRunningTime="2026-10-30 18:18:25.157237615 +0000 UTC m=+1055.977949392" Oct 30 18:18:31.118508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:31.118478 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:18:40.717279 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:18:40.717232 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:18:41.803342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:41.803294 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.261321 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:42.261269 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.311457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:42.310215 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.479795 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:42.479730 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.795222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:42.792322 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 18:18:43.263738 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.263690 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.273657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.265039 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:18:43.274026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.273987 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:18:43.317652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.317574 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.368969 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.368924 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.375052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.375019 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:43.377617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.377564 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:43.520797 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.520705 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 18:18:43.521367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.521347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:18:43.595747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.595716 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:18:43.675448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.674194 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:18:43.805050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.804943 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.805871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:43.805845 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:18:44.173530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.170902 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.278293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.278236 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.364857 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.364511 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:44.479506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.479117 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:18:44.549183 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.545646 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.796910 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.796801 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 18:18:44.798184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.798158 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:18:44.817625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:44.810851 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.066519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.066409 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:18:45.139823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.139774 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:18:45.180703 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.175975 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.184858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.184516 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:18:45.242838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.242749 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.373690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.365972 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.502619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.493676 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 18:18:45.558378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.558328 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.594624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.594083 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.605016 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.603257 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.821624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.821211 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:18:45.919622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:45.918700 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.377623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:46.375065 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.377623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:46.375631 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:46.599942 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:46.599910 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:18:46.715625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:46.702697 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.801623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:46.801394 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:18:53.777213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.777176 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:18:53.785459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.785435 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.804209 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.804170 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:18:53.806266 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.806233 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:18:53.806561 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.806539 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:18:53.806782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.806755 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:18:53.807064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.807046 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" Oct 30 18:18:53.889800 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.889759 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890018 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.889836 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890018 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.889965 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.890039 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.890066 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.890247 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6wz\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.890316 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.890303 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992055 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992174 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992252 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992438 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6wz\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.992890 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.992863 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.993385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.993067 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.993385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.993038 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.995042 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.995013 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.995397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.995373 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.995829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.995809 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:53.995932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:53.995911 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:54.012465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:54.012432 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:54.012659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:54.012536 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6wz\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xttzf\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:54.113047 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:54.111895 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:54.529971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:54.529935 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:18:54.533237 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:54.533206 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5e8b1e_d5f1_48d7_94d1_442d974b7093.slice/crio-ee0061fc1367623d7ffd0fed9e16e542111832a8ad9bc3ec83d6b487263b671a WatchSource:0}: Error finding container ee0061fc1367623d7ffd0fed9e16e542111832a8ad9bc3ec83d6b487263b671a: Status 404 returned error can't find the container with id ee0061fc1367623d7ffd0fed9e16e542111832a8ad9bc3ec83d6b487263b671a Oct 30 18:18:55.355618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:55.355567 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" event={"ID":"1f5e8b1e-d5f1-48d7-94d1-442d974b7093","Type":"ContainerStarted","Data":"ee0061fc1367623d7ffd0fed9e16e542111832a8ad9bc3ec83d6b487263b671a"} Oct 30 18:18:56.265231 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.265189 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:18:56.274615 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.274567 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342464 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342525 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342688 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342718 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342839 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342884 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342911 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.342998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.342992 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.343210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.343031 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5kw\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.353499 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.353466 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:18:56.378254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.378221 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7cb9c50e-openshift-default-dockercfg-rvfbm\"" Oct 30 18:18:56.445179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5kw\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.445365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445294 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.445365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445336 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.445550 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445529 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.445588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445562 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.445690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.445676 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446032 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446070 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446092 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446107 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446281 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446148 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446290 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446687 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.446945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.446924 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.448394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.448370 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.448965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.448944 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.463168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.463132 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.463304 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.463252 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5kw\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.615381 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.615291 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:56.988646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.988612 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:18:56.991499 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:18:56.991453 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416d589c_6c2b_4e3d_b1ce_c4af0cc4f510.slice/crio-168260d0ff9f52e0d1fb7e841cc19bf8cbe22ef9873e0409eab559d6e2c62bba WatchSource:0}: Error finding container 168260d0ff9f52e0d1fb7e841cc19bf8cbe22ef9873e0409eab559d6e2c62bba: Status 404 returned error can't find the container with id 168260d0ff9f52e0d1fb7e841cc19bf8cbe22ef9873e0409eab559d6e2c62bba Oct 30 18:18:56.995383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.995342 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:56.995486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.995445 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:56.995559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:56.995499 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.230311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.230273 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.230418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.230354 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:18:57.382952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.382914 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" event={"ID":"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510","Type":"ContainerStarted","Data":"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9"} Oct 30 18:18:57.383528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.383502 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" event={"ID":"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510","Type":"ContainerStarted","Data":"168260d0ff9f52e0d1fb7e841cc19bf8cbe22ef9873e0409eab559d6e2c62bba"} Oct 30 18:18:57.385570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.385547 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" event={"ID":"1f5e8b1e-d5f1-48d7-94d1-442d974b7093","Type":"ContainerStarted","Data":"2b0f299dfc139d73ac9776678b1f6da1fd89260c2733244fe821b7e94e39e2f4"} Oct 30 18:18:57.385832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.385812 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:18:57.422564 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.422500 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podStartSLOduration=1.422478991 podStartE2EDuration="1.422478991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:57.420092705 +0000 UTC m=+1088.240804481" watchObservedRunningTime="2026-10-30 18:18:57.422478991 +0000 UTC m=+1088.243190767" Oct 30 18:18:57.458995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.458838 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" podStartSLOduration=1.7666155300000002 podStartE2EDuration="4.458800224s" totalImagesPullingTime="2.692184694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:57.455659285 +0000 UTC m=+1088.276371062" watchObservedRunningTime="2026-10-30 18:18:57.458800224 +0000 UTC m=+1088.279511999" Oct 30 18:18:57.615558 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:57.615474 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:18:58.403735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:58.403695 5024 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xttzf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:18:58.404417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:58.404385 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:18:58.654674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:58.648708 5024 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:18:58.675619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:58.660758 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 18:18:59.616568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:59.616531 5024 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.617026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:18:59.616589 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 18:19:00.615685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:00.615652 5024 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.615852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:00.615742 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 18:19:01.399397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:01.399364 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:19:01.616524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:01.616492 5024 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.616729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:01.616548 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 18:19:02.616368 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:02.616335 5024 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:19:02.616790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:02.616386 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 18:19:03.621232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:03.621197 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:19:03.621680 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:03.621630 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:19:03.622483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:03.622457 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:19:07.947880 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:07.947841 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:19:10.441054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:10.441009 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:19:10.441701 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:10.441239 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" podUID="919d62aa-b58d-403e-aadf-794515ddc80b" containerName="maas-api" containerID="cri-o://8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e" gracePeriod=30 Oct 30 18:19:10.492342 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:10.492299 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:19:10.936903 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:10.936866 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:19:11.017865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.017762 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\") pod \"919d62aa-b58d-403e-aadf-794515ddc80b\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " Oct 30 18:19:11.018067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.017933 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\") pod \"919d62aa-b58d-403e-aadf-794515ddc80b\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " Oct 30 18:19:11.018067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.018006 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\") pod \"919d62aa-b58d-403e-aadf-794515ddc80b\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " Oct 30 18:19:11.018067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.018033 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\" (UniqueName: \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\") pod \"919d62aa-b58d-403e-aadf-794515ddc80b\" (UID: \"919d62aa-b58d-403e-aadf-794515ddc80b\") " Oct 30 18:19:11.019006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.018974 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca" pod "919d62aa-b58d-403e-aadf-794515ddc80b" (UID: "919d62aa-b58d-403e-aadf-794515ddc80b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:11.020876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.020849 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls" pod "919d62aa-b58d-403e-aadf-794515ddc80b" (UID: "919d62aa-b58d-403e-aadf-794515ddc80b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:11.035351 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.035295 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs" pod "919d62aa-b58d-403e-aadf-794515ddc80b" (UID: "919d62aa-b58d-403e-aadf-794515ddc80b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:11.035519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.035446 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45" pod "919d62aa-b58d-403e-aadf-794515ddc80b" (UID: "919d62aa-b58d-403e-aadf-794515ddc80b"). InnerVolumeSpecName "kube-api-access-x7j45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:11.120784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.120747 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:11.120784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.120783 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/919d62aa-b58d-403e-aadf-794515ddc80b-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:11.121019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.120796 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/919d62aa-b58d-403e-aadf-794515ddc80b-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:11.121019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.120808 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7j45\" (UniqueName: \"kubernetes.io/projected/919d62aa-b58d-403e-aadf-794515ddc80b-kube-api-access-x7j45\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:11.515896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.515405 5024 generic.go:356] "Generic (PLEG): container finished" podID="919d62aa-b58d-403e-aadf-794515ddc80b" containerID="8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e" exitCode=0 Oct 30 18:19:11.515896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.515685 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" event={"ID":"919d62aa-b58d-403e-aadf-794515ddc80b","Type":"ContainerDied","Data":"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e"} Oct 30 18:19:11.515896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.515734 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" event={"ID":"919d62aa-b58d-403e-aadf-794515ddc80b","Type":"ContainerDied","Data":"d45f2a27881eec26ac290fe602885a7d0f1a2ee5b09803795d9cde4d7b2817c6"} Oct 30 18:19:11.515896 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.515763 5024 scope.go:122] "RemoveContainer" containerID="8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e" Oct 30 18:19:11.516389 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.515902 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82" Oct 30 18:19:11.547490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.547460 5024 scope.go:122] "RemoveContainer" containerID="8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e" Oct 30 18:19:11.550442 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:11.550361 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e\": container with ID starting with 8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e not found: ID does not exist" containerID="8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e" Oct 30 18:19:11.550442 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.550407 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e"} err="failed to get container status \"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e\": rpc error: code = NotFound desc = could not find container \"8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e\": container with ID starting with 8c8f93a62378d1fbfbeeacbcea09886dfde532d38f0d222b3dce2832ad9a453e not found: ID does not exist" Oct 30 18:19:11.615675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.602678 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:19:11.615675 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:11.614676 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-535b743b-6b5f5dcf95-ctr82"] Oct 30 18:19:12.948710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:12.948652 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" podUID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" containerName="payload-pre-processing" containerID="cri-o://7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265" gracePeriod=25 Oct 30 18:19:13.285930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.285904 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:19:13.308252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.308216 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919d62aa-b58d-403e-aadf-794515ddc80b" path="/var/lib/kubelet/pods/919d62aa-b58d-403e-aadf-794515ddc80b/volumes" Oct 30 18:19:13.373433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.373392 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\" (UniqueName: \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\") pod \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " Oct 30 18:19:13.373688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.373561 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\") pod \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\" (UID: \"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96\") " Oct 30 18:19:13.374704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.374674 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume" pod "8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" (UID: "8fb1828b-92ad-4bee-b7b1-eb041cbe3c96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:13.377724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.377693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh" pod "8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" (UID: "8fb1828b-92ad-4bee-b7b1-eb041cbe3c96"). InnerVolumeSpecName "kube-api-access-d9dnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:13.475826 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.475752 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dnh\" (UniqueName: \"kubernetes.io/projected/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-kube-api-access-d9dnh\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:13.475826 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.475783 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96-plugins-config-volume\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:13.575865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.575779 5024 generic.go:356] "Generic (PLEG): container finished" podID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" containerID="7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265" exitCode=0 Oct 30 18:19:13.576042 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.575989 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" event={"ID":"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96","Type":"ContainerDied","Data":"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265"} Oct 30 18:19:13.576128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.576043 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" event={"ID":"8fb1828b-92ad-4bee-b7b1-eb041cbe3c96","Type":"ContainerDied","Data":"a6afc19230d0afa133859ec4e04377059d7b0dd83dd3ea26bcca104c3d4a4de1"} Oct 30 18:19:13.576128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.576074 5024 scope.go:122] "RemoveContainer" containerID="7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265" Oct 30 18:19:13.576388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.576239 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k" Oct 30 18:19:13.594418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.594390 5024 scope.go:122] "RemoveContainer" containerID="7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265" Oct 30 18:19:13.594921 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:13.594892 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265\": container with ID starting with 7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265 not found: ID does not exist" containerID="7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265" Oct 30 18:19:13.595052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.594929 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265"} err="failed to get container status \"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265\": rpc error: code = NotFound desc = could not find container \"7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265\": container with ID starting with 7a468886c91689c3a5935419ec2dcf9324b5a0f10f92a05419c2b3c67ed20265 not found: ID does not exist" Oct 30 18:19:13.635344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.635306 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:19:13.639874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.639842 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7cb9c50e-54c9cdc579-qv74k"] Oct 30 18:19:13.765839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.765807 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:19:13.808791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:13.808749 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:19:15.307341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:15.307309 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" path="/var/lib/kubelet/pods/8fb1828b-92ad-4bee-b7b1-eb041cbe3c96/volumes" Oct 30 18:19:17.330514 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.330472 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:19:17.332066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.330782 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" podUID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" containerName="maas-api" containerID="cri-o://5691e9d27021b03ff959f999806e2cac61e0ab02d154a9e84162a7f5858ae763" gracePeriod=30 Oct 30 18:19:17.695968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.695935 5024 generic.go:356] "Generic (PLEG): container finished" podID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" containerID="5691e9d27021b03ff959f999806e2cac61e0ab02d154a9e84162a7f5858ae763" exitCode=0 Oct 30 18:19:17.696243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.696226 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" event={"ID":"c99dea5e-f41d-4c76-8611-ae8e0b1985bd","Type":"ContainerDied","Data":"5691e9d27021b03ff959f999806e2cac61e0ab02d154a9e84162a7f5858ae763"} Oct 30 18:19:17.719120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.719085 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:19:17.798393 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.798355 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\" (UniqueName: \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\") pod \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " Oct 30 18:19:17.798702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.798433 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\") pod \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " Oct 30 18:19:17.798702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.798625 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\") pod \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " Oct 30 18:19:17.798886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.798852 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\") pod \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\" (UID: \"c99dea5e-f41d-4c76-8611-ae8e0b1985bd\") " Oct 30 18:19:17.800886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.800850 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca" pod "c99dea5e-f41d-4c76-8611-ae8e0b1985bd" (UID: "c99dea5e-f41d-4c76-8611-ae8e0b1985bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:17.840676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.817794 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7" pod "c99dea5e-f41d-4c76-8611-ae8e0b1985bd" (UID: "c99dea5e-f41d-4c76-8611-ae8e0b1985bd"). InnerVolumeSpecName "kube-api-access-h92w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:17.840676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.820582 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs" pod "c99dea5e-f41d-4c76-8611-ae8e0b1985bd" (UID: "c99dea5e-f41d-4c76-8611-ae8e0b1985bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:17.853273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.851172 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls" pod "c99dea5e-f41d-4c76-8611-ae8e0b1985bd" (UID: "c99dea5e-f41d-4c76-8611-ae8e0b1985bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:17.901771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.901730 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.901981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.901772 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h92w7\" (UniqueName: \"kubernetes.io/projected/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-kube-api-access-h92w7\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.901981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.901800 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.901981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:17.901814 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99dea5e-f41d-4c76-8611-ae8e0b1985bd-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:18.704702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.704671 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" Oct 30 18:19:18.705119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.704671 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb" event={"ID":"c99dea5e-f41d-4c76-8611-ae8e0b1985bd","Type":"ContainerDied","Data":"56372292ed33e3c2f9f3909e003da8bc92ed3ff2dc71596c0c14a5993d8a3009"} Oct 30 18:19:18.705119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.704803 5024 scope.go:122] "RemoveContainer" containerID="5691e9d27021b03ff959f999806e2cac61e0ab02d154a9e84162a7f5858ae763" Oct 30 18:19:18.749731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.749698 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:19:18.760493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.760461 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7cb9c50e-76b7cd58c7-cvpzb"] Oct 30 18:19:18.766087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.766043 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" podUID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" containerName="payload-processing" containerID="cri-o://34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7" gracePeriod=25 Oct 30 18:19:18.811646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.809401 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" podUID="9c207332-4822-42fc-a08e-420916c949d2" containerName="payload-pre-processing" containerID="cri-o://3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d" gracePeriod=25 Oct 30 18:19:18.855309 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.855262 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:19:18.856855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:18.856557 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" containerID="cri-o://1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9" gracePeriod=30 Oct 30 18:19:19.260909 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.260879 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:19:19.315848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.315758 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" path="/var/lib/kubelet/pods/c99dea5e-f41d-4c76-8611-ae8e0b1985bd/volumes" Oct 30 18:19:19.335869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.333807 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\") pod \"9c207332-4822-42fc-a08e-420916c949d2\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " Oct 30 18:19:19.335869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.333860 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\" (UniqueName: \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\") pod \"9c207332-4822-42fc-a08e-420916c949d2\" (UID: \"9c207332-4822-42fc-a08e-420916c949d2\") " Oct 30 18:19:19.335869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.334573 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume" pod "9c207332-4822-42fc-a08e-420916c949d2" (UID: "9c207332-4822-42fc-a08e-420916c949d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:19.335869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.335586 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c207332-4822-42fc-a08e-420916c949d2-plugins-config-volume\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:19.344797 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.344771 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2" pod "9c207332-4822-42fc-a08e-420916c949d2" (UID: "9c207332-4822-42fc-a08e-420916c949d2"). InnerVolumeSpecName "kube-api-access-8cps2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:19.386924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.386888 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:19:19.437270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.437226 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cps2\" (UniqueName: \"kubernetes.io/projected/9c207332-4822-42fc-a08e-420916c949d2-kube-api-access-8cps2\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:19.538699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.538656 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\" (UniqueName: \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\") pod \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " Oct 30 18:19:19.538872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.538818 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\") pod \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\" (UID: \"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a\") " Oct 30 18:19:19.539275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.539242 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume" pod "3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" (UID: "3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:19.539773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.539738 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-plugins-config-volume\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:19.541383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.541356 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5" pod "3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" (UID: "3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a"). InnerVolumeSpecName "kube-api-access-l9mp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:19.641299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.641270 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mp5\" (UniqueName: \"kubernetes.io/projected/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a-kube-api-access-l9mp5\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:19.714134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.714096 5024 generic.go:356] "Generic (PLEG): container finished" podID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" containerID="34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7" exitCode=0 Oct 30 18:19:19.714939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.714147 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" Oct 30 18:19:19.714939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.714185 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" event={"ID":"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a","Type":"ContainerDied","Data":"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7"} Oct 30 18:19:19.714939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.714244 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z" event={"ID":"3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a","Type":"ContainerDied","Data":"aa151d81e16b9e9091c89898c02aa2bee618ee5c0e70315a5d242d8ec4f076e4"} Oct 30 18:19:19.714939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.714270 5024 scope.go:122] "RemoveContainer" containerID="34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7" Oct 30 18:19:19.717884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.717853 5024 generic.go:356] "Generic (PLEG): container finished" podID="9c207332-4822-42fc-a08e-420916c949d2" containerID="3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d" exitCode=0 Oct 30 18:19:19.718740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.717973 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" event={"ID":"9c207332-4822-42fc-a08e-420916c949d2","Type":"ContainerDied","Data":"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d"} Oct 30 18:19:19.718740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.718024 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" event={"ID":"9c207332-4822-42fc-a08e-420916c949d2","Type":"ContainerDied","Data":"7b39e84b427f33664efb0a6c052dad9cab6faecf190004e80f8ed0e158740276"} Oct 30 18:19:19.718740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.718100 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt" Oct 30 18:19:19.731827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.731740 5024 scope.go:122] "RemoveContainer" containerID="34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7" Oct 30 18:19:19.732413 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:19.732385 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7\": container with ID starting with 34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7 not found: ID does not exist" containerID="34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7" Oct 30 18:19:19.732694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.732656 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7"} err="failed to get container status \"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7\": rpc error: code = NotFound desc = could not find container \"34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7\": container with ID starting with 34d7c691b925784809459b13fe90b7609c300681074b88583da9c93258d0f1e7 not found: ID does not exist" Oct 30 18:19:19.732694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.732694 5024 scope.go:122] "RemoveContainer" containerID="3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d" Oct 30 18:19:19.763869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.763845 5024 scope.go:122] "RemoveContainer" containerID="3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d" Oct 30 18:19:19.764253 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:19.764224 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d\": container with ID starting with 3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d not found: ID does not exist" containerID="3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d" Oct 30 18:19:19.764388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.764266 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d"} err="failed to get container status \"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d\": rpc error: code = NotFound desc = could not find container \"3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d\": container with ID starting with 3c09113249347fc6bb8e48d947dc8fb9168076f02aa4d3c8aff2d7e355004f0d not found: ID does not exist" Oct 30 18:19:19.794354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.794315 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:19:19.805200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.805169 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f6a224e7-76488bf5c5-2khrt"] Oct 30 18:19:19.852777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.852738 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:19:19.856240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:19.856211 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f6a224e7-6bfbfdf8d4-kg99z"] Oct 30 18:19:20.612235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.612200 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:19:20.612976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.612954 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9c207332-4822-42fc-a08e-420916c949d2" containerName="payload-pre-processing" Oct 30 18:19:20.612976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.612977 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c207332-4822-42fc-a08e-420916c949d2" containerName="payload-pre-processing" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613030 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" containerName="maas-api" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613040 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" containerName="maas-api" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613054 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" containerName="payload-pre-processing" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613061 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" containerName="payload-pre-processing" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613085 5024 cpu_manager.go:414] "containerMap: removing container" podUID="919d62aa-b58d-403e-aadf-794515ddc80b" containerName="maas-api" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613092 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919d62aa-b58d-403e-aadf-794515ddc80b" containerName="maas-api" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613112 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" containerName="payload-processing" Oct 30 18:19:20.613160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613121 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" containerName="payload-processing" Oct 30 18:19:20.613547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613302 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb1828b-92ad-4bee-b7b1-eb041cbe3c96" containerName="payload-pre-processing" Oct 30 18:19:20.613547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613322 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="919d62aa-b58d-403e-aadf-794515ddc80b" containerName="maas-api" Oct 30 18:19:20.613547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613336 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" containerName="payload-processing" Oct 30 18:19:20.613547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613351 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c207332-4822-42fc-a08e-420916c949d2" containerName="payload-pre-processing" Oct 30 18:19:20.613547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.613366 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99dea5e-f41d-4c76-8611-ae8e0b1985bd" containerName="maas-api" Oct 30 18:19:20.618953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.618930 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.642186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.642145 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b2b596-serving-cert\"" Oct 30 18:19:20.648512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.648485 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:19:20.772208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.772173 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.772671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.772365 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsltc\" (UniqueName: \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.772735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.772680 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.772798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.772780 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.874443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.874344 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.874443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.874405 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.874711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.874448 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.874711 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:20.874520 5024 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-serving-cert: secret "maas-api-e2e-shared-a-w3-b2b596-serving-cert" not found Oct 30 18:19:20.874711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.874529 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsltc\" (UniqueName: \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.874711 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:20.874679 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls podName:0580522a-0b43-402b-838a-1896f1ffca93 nodeName:}" failed. No retries permitted until 2026-10-30 18:19:21.37465673 +0000 UTC m=+1112.195368686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls") pod "maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" (UID: "0580522a-0b43-402b-838a-1896f1ffca93") : secret "maas-api-e2e-shared-a-w3-b2b596-serving-cert" not found Oct 30 18:19:20.875194 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.875167 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.877205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.877188 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:20.892377 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:20.892346 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsltc\" (UniqueName: \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:21.306660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.306620 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a" path="/var/lib/kubelet/pods/3d15a2a0-f6cb-4d9b-a449-2b2c6910a78a/volumes" Oct 30 18:19:21.307169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.307152 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c207332-4822-42fc-a08e-420916c949d2" path="/var/lib/kubelet/pods/9c207332-4822-42fc-a08e-420916c949d2/volumes" Oct 30 18:19:21.382593 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.382541 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:21.385233 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.385208 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:21.537161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.536901 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:21.951643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:21.941300 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:19:22.751493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:22.751449 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" event={"ID":"0580522a-0b43-402b-838a-1896f1ffca93","Type":"ContainerStarted","Data":"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0"} Oct 30 18:19:22.751739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:22.751721 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:22.751843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:22.751771 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" event={"ID":"0580522a-0b43-402b-838a-1896f1ffca93","Type":"ContainerStarted","Data":"b853ef01a3fb0b7b158a67f73e6bccafc3ca82d1beb81ad82773ab3c76fdbde3"} Oct 30 18:19:22.789398 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:22.789327 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" podStartSLOduration=2.428415811 podStartE2EDuration="2.789303577s" totalImagesPullingTime="360.887766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:22.785077766 +0000 UTC m=+1113.605789546" watchObservedRunningTime="2026-10-30 18:19:22.789303577 +0000 UTC m=+1113.610015354" Oct 30 18:19:24.307473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.307443 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357129 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357277 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357323 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357434 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357477 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.357622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357553 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.358060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357722 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.358060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357759 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.358060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.357793 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\") pod \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\" (UID: \"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510\") " Oct 30 18:19:24.373862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.373101 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.373862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.373397 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:24.373862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.373771 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.373862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.373836 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.374582 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.374549 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.379207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.379168 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "kube-api-access-nt5kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:24.389053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.384788 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.389053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.384849 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:24.389053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.384996 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token" pod "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" (UID: "416d589c-6c2b-4e3d-b1ce-c4af0cc4f510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:24.463087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.462979 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.463721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.463637 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.463916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.463893 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5kw\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-kube-api-access-nt5kw\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.463956 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.463976 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.463990 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.464009 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.464024 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.464139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.464038 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.769374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.769334 5024 generic.go:356] "Generic (PLEG): container finished" podID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerID="1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9" exitCode=0 Oct 30 18:19:24.769653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.769400 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" event={"ID":"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510","Type":"ContainerDied","Data":"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9"} Oct 30 18:19:24.769653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.769446 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" Oct 30 18:19:24.769653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.769482 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892" event={"ID":"416d589c-6c2b-4e3d-b1ce-c4af0cc4f510","Type":"ContainerDied","Data":"168260d0ff9f52e0d1fb7e841cc19bf8cbe22ef9873e0409eab559d6e2c62bba"} Oct 30 18:19:24.769653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.769515 5024 scope.go:122] "RemoveContainer" containerID="1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9" Oct 30 18:19:24.785113 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.785036 5024 scope.go:122] "RemoveContainer" containerID="1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9" Oct 30 18:19:24.785482 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:24.785458 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9\": container with ID starting with 1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9 not found: ID does not exist" containerID="1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9" Oct 30 18:19:24.785547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.785494 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9"} err="failed to get container status \"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9\": rpc error: code = NotFound desc = could not find container \"1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9\": container with ID starting with 1f4df7f62ea59b80792d07a1d6d0feb297b41f433b308d4a94166937b1fb29f9 not found: ID does not exist" Oct 30 18:19:24.818816 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.818784 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:19:24.827679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:24.827644 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bn892"] Oct 30 18:19:25.318168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:25.318135 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" path="/var/lib/kubelet/pods/416d589c-6c2b-4e3d-b1ce-c4af0cc4f510/volumes" Oct 30 18:19:28.764789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:19:28.764688 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:19:40.458282 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:19:40.458237 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:20:10.579923 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:20:10.579561 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:20:26.918537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.918503 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:20:26.919425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.919405 5024 cpu_manager.go:414] "containerMap: removing container" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" Oct 30 18:20:26.919477 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.919426 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" Oct 30 18:20:26.919566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.919524 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="416d589c-6c2b-4e3d-b1ce-c4af0cc4f510" containerName="istio-proxy" Oct 30 18:20:26.923227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.923209 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:26.951434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:26.951404 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:20:27.029743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.029700 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" Oct 30 18:20:27.053630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053584 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xhk\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.053806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053639 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.053806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053754 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.053806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053793 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.053967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053867 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.053967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.053928 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.054037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.054027 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.054107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.054093 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.054148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.054137 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155456 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xhk\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155498 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155547 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155583 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155650 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155693 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155766 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155844 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155919 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.155984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.155944 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.156214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.156184 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.156276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.156238 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.156334 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.156294 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.156848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.156826 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.157945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.157924 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.158312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.158276 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.181877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.181817 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xhk\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.183480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.183461 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.240499 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.240465 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:27.581791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.581765 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:20:27.583578 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:20:27.583552 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11eb5e2b_31d5_46a4_8482_09fc9691a50c.slice/crio-57aa6cf7bf0e6a767da6e407599c97df08a07c29abcd1577ad0b44eb52f32623 WatchSource:0}: Error finding container 57aa6cf7bf0e6a767da6e407599c97df08a07c29abcd1577ad0b44eb52f32623: Status 404 returned error can't find the container with id 57aa6cf7bf0e6a767da6e407599c97df08a07c29abcd1577ad0b44eb52f32623 Oct 30 18:20:27.586351 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.586315 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:27.586454 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.586400 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:27.586454 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:27.586445 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:28.476083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:28.476032 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" event={"ID":"11eb5e2b-31d5-46a4-8482-09fc9691a50c","Type":"ContainerStarted","Data":"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3"} Oct 30 18:20:28.476514 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:28.476098 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" event={"ID":"11eb5e2b-31d5-46a4-8482-09fc9691a50c","Type":"ContainerStarted","Data":"57aa6cf7bf0e6a767da6e407599c97df08a07c29abcd1577ad0b44eb52f32623"} Oct 30 18:20:29.241254 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:29.241225 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:29.242584 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:29.242561 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:29.242727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:29.242640 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 18:20:30.241702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:30.241662 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:30.242159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:30.241716 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 18:20:31.241819 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:31.241781 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:31.242200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:31.241828 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 18:20:32.241316 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:32.241280 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:32.241497 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:32.241333 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 18:20:33.241077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:33.241042 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:20:33.241571 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:33.241094 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 18:20:34.245097 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:34.245068 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:34.245550 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:34.245537 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:34.246295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:34.246275 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:20:34.292552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:34.292490 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podStartSLOduration=8.292473203 podStartE2EDuration="8.292473203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:28.531627576 +0000 UTC m=+1179.352339343" watchObservedRunningTime="2026-10-30 18:20:34.292473203 +0000 UTC m=+1185.113184980" Oct 30 18:20:40.455812 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:20:40.455777 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:20:42.315293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.315258 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:20:42.319885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.319863 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.343551 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.343519 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ae3373-serving-cert\"" Oct 30 18:20:42.347794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.347769 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:20:42.385554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.385508 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnph\" (UniqueName: \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.386242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.386209 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.386362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.386301 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.386434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.386409 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.487914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.487884 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnph\" (UniqueName: \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.488111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.487976 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.488111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.488035 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.488111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.488068 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.488854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.488777 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.490745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.490721 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.490854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.490780 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.504756 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.504733 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnph\" (UniqueName: \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\") pod \"maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.634661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.634551 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:42.973618 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:20:42.973553 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11823105_6dc6_4a83_b3fc_ccc7d9d4c740.slice/crio-09a832a7cb1101e58c8f7c995b66aa265eda58f7e3793ba60f26b8364bd5f3b4 WatchSource:0}: Error finding container 09a832a7cb1101e58c8f7c995b66aa265eda58f7e3793ba60f26b8364bd5f3b4: Status 404 returned error can't find the container with id 09a832a7cb1101e58c8f7c995b66aa265eda58f7e3793ba60f26b8364bd5f3b4 Oct 30 18:20:42.973772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:42.973661 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:20:43.613155 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:43.613116 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" event={"ID":"11823105-6dc6-4a83-b3fc-ccc7d9d4c740","Type":"ContainerStarted","Data":"5df315336e7f2d67bf3fe77299b9f84fadc42594df624524829f3b70af79c996"} Oct 30 18:20:43.613608 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:43.613216 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" event={"ID":"11823105-6dc6-4a83-b3fc-ccc7d9d4c740","Type":"ContainerStarted","Data":"09a832a7cb1101e58c8f7c995b66aa265eda58f7e3793ba60f26b8364bd5f3b4"} Oct 30 18:20:43.613608 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:43.613371 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:43.651650 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:43.651502 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" podStartSLOduration=1.235639976 podStartE2EDuration="1.651487323s" totalImagesPullingTime="415.847347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:43.648062498 +0000 UTC m=+1194.468774274" watchObservedRunningTime="2026-10-30 18:20:43.651487323 +0000 UTC m=+1194.472199099" Oct 30 18:20:45.622888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.622856 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t4rcs-7z9m4"] Oct 30 18:20:45.627474 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.627455 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.642463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.642435 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t4rcs-7z9m4"] Oct 30 18:20:45.758785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.758737 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.758982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.758898 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8mv\" (UniqueName: \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.759091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.759077 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.759208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.759190 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.860378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.860343 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.860548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.860413 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.860548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.860470 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8mv\" (UniqueName: \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.860743 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.860726 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.861057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.861035 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.861149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.861080 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.861204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.861186 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.878641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.878552 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8mv\" (UniqueName: \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\") pod \"periodic-gathering-t4rcs-7z9m4\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:45.945274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:45.945228 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:20:46.263492 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:46.263460 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t4rcs-7z9m4"] Oct 30 18:20:46.267093 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:20:46.267064 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760dff11_9ff5_4f46_a8e9_b34637ba2053.slice/crio-6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60 WatchSource:0}: Error finding container 6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60: Status 404 returned error can't find the container with id 6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60 Oct 30 18:20:46.638944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:46.638847 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" event={"ID":"760dff11-9ff5-4f46-a8e9-b34637ba2053","Type":"ContainerStarted","Data":"52569f0dac0805894e57902963bb1add7018cf43b868338ded3b70ab6b0dc06f"} Oct 30 18:20:46.638944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:46.638915 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" event={"ID":"760dff11-9ff5-4f46-a8e9-b34637ba2053","Type":"ContainerStarted","Data":"6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60"} Oct 30 18:20:46.674561 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:46.674505 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" podStartSLOduration=1.6744888329999998 podStartE2EDuration="1.674488833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:46.67090206 +0000 UTC m=+1197.491613836" watchObservedRunningTime="2026-10-30 18:20:46.674488833 +0000 UTC m=+1197.495200608" Oct 30 18:20:49.487748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:49.487717 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:20:49.490830 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:49.490809 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:20:49.626432 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:49.626403 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:20:57.014096 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.014062 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:20:57.018807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.018765 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.039150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.039114 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:20:57.063166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063125 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063192 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063262 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063303 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063330 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063359 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7qv\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063623 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063657 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.063735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.063687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.164768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.164721 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.164768 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.164786 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.164895 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.164948 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7qv\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165071 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165128 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165209 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165217 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165428 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165336 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165534 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.165686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.165663 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.166221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.166185 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.167874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.167846 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.167989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.167912 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.183465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.183431 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.183612 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.183447 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7qv\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.333659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.333575 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:57.661300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.661271 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:20:57.662757 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:20:57.662726 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd744c62_c781_4a12_8c9d_e789b39e50cd.slice/crio-f46af2e66666053d2d6a8107b96121495d54f0e8f5aa1fd279d4b170cdab4383 WatchSource:0}: Error finding container f46af2e66666053d2d6a8107b96121495d54f0e8f5aa1fd279d4b170cdab4383: Status 404 returned error can't find the container with id f46af2e66666053d2d6a8107b96121495d54f0e8f5aa1fd279d4b170cdab4383 Oct 30 18:20:57.665630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.665577 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:57.665714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.665665 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:57.665714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.665702 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:57.848316 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.848281 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" event={"ID":"cd744c62-c781-4a12-8c9d-e789b39e50cd","Type":"ContainerStarted","Data":"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514"} Oct 30 18:20:57.848528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.848336 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" event={"ID":"cd744c62-c781-4a12-8c9d-e789b39e50cd","Type":"ContainerStarted","Data":"f46af2e66666053d2d6a8107b96121495d54f0e8f5aa1fd279d4b170cdab4383"} Oct 30 18:20:57.891572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:57.891519 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podStartSLOduration=1.891504691 podStartE2EDuration="1.891504691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:57.890375685 +0000 UTC m=+1208.711087477" watchObservedRunningTime="2026-10-30 18:20:57.891504691 +0000 UTC m=+1208.712216467" Oct 30 18:20:58.334964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:58.334720 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:20:58.336043 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:58.336017 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:20:58.336195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:58.336071 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:20:59.336120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:59.336074 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:20:59.336660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:20:59.336153 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:21:00.334714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:00.334683 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:21:00.334910 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:00.334737 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:21:01.334964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:01.334927 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:21:01.335412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:01.335001 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:21:02.333959 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:02.333919 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:21:02.334144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:02.333974 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:21:03.334799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:03.334762 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:21:03.335236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:03.334826 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 18:21:04.339050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:04.339020 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:21:04.339498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:04.339479 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:21:04.340257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:04.340237 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:21:10.458155 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:21:10.458104 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:21:27.728794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.728757 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:21:27.735099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.735072 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:27.750304 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.750270 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:21:27.754074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.754042 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e5d168\"" Oct 30 18:21:27.762587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.762560 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e5d168-dockercfg-hlfd9\"" Oct 30 18:21:27.908469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.907848 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8q8d\" (UniqueName: \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:27.908469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:27.907963 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.015974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.015858 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8q8d\" (UniqueName: \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.015974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.015944 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.017963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.017108 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.036653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.034496 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8q8d\" (UniqueName: \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\") pod \"payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.053525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.053485 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:28.370902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:28.370876 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:21:28.373231 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:21:28.373197 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd90bf0_2165_4e1b_9d1f_76b616c036ed.slice/crio-2f5448085c2a8c202cb2fcfef10047bef45b999ae062d6fab638c46640c558c0 WatchSource:0}: Error finding container 2f5448085c2a8c202cb2fcfef10047bef45b999ae062d6fab638c46640c558c0: Status 404 returned error can't find the container with id 2f5448085c2a8c202cb2fcfef10047bef45b999ae062d6fab638c46640c558c0 Oct 30 18:21:29.117523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:29.117487 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" event={"ID":"0fd90bf0-2165-4e1b-9d1f-76b616c036ed","Type":"ContainerStarted","Data":"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2"} Oct 30 18:21:29.118125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:29.117539 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" event={"ID":"0fd90bf0-2165-4e1b-9d1f-76b616c036ed","Type":"ContainerStarted","Data":"2f5448085c2a8c202cb2fcfef10047bef45b999ae062d6fab638c46640c558c0"} Oct 30 18:21:29.118125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:29.117633 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:29.151939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:29.151876 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" podStartSLOduration=2.151863276 podStartE2EDuration="2.151863276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:29.151429533 +0000 UTC m=+1239.972141310" watchObservedRunningTime="2026-10-30 18:21:29.151863276 +0000 UTC m=+1239.972575054" Oct 30 18:21:36.242490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.242452 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:21:36.247175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.247149 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.262390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.262344 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"openshift-service-ca.crt\"" Oct 30 18:21:36.262390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.262344 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"kube-root-ca.crt\"" Oct 30 18:21:36.262590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.262563 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:21:36.269998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.269969 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"e2e-lifecycle-model-fb0abbf7-kserve-self-signed-certs\"" Oct 30 18:21:36.270147 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.269972 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"default-dockercfg-52wg8\"" Oct 30 18:21:36.337137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337091 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.337359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337334 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.337447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337424 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9t29\" (UniqueName: \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.337505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337493 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.337568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337535 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.337680 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.337663 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440030 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440281 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440147 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9t29\" (UniqueName: \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440281 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440185 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440281 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440225 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440437 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440554 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440621 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.440840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.440818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.442818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.442770 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.442981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.442967 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.458643 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.458608 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9t29\" (UniqueName: \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\") pod \"e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.565411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.565312 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:36.929552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:36.929521 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:21:36.932238 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:21:36.932205 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff46dbfd_1e57_47fc_ae2b_c5d3920cf411.slice/crio-f9bacf4fa6cb996bde9dcb24561fc1d33aabd315048dc0b23d25d1173ee0fe33 WatchSource:0}: Error finding container f9bacf4fa6cb996bde9dcb24561fc1d33aabd315048dc0b23d25d1173ee0fe33: Status 404 returned error can't find the container with id f9bacf4fa6cb996bde9dcb24561fc1d33aabd315048dc0b23d25d1173ee0fe33 Oct 30 18:21:37.192056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:37.191962 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" event={"ID":"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411","Type":"ContainerStarted","Data":"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623"} Oct 30 18:21:37.192056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:37.192008 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" event={"ID":"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411","Type":"ContainerStarted","Data":"f9bacf4fa6cb996bde9dcb24561fc1d33aabd315048dc0b23d25d1173ee0fe33"} Oct 30 18:21:37.230069 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:37.230014 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" podStartSLOduration=1.229998741 podStartE2EDuration="1.229998741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:37.225842281 +0000 UTC m=+1248.046554050" watchObservedRunningTime="2026-10-30 18:21:37.229998741 +0000 UTC m=+1248.050710515" Oct 30 18:21:37.566970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:37.566870 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:40.130222 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:40.130191 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:21:40.458186 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:21:40.458144 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:21:46.044294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.044260 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:21:46.048957 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.048937 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.065146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.065115 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:21:46.069662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.069584 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e5d168-serving-cert\"" Oct 30 18:21:46.212125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.212087 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.212307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.212291 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.212364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.212344 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh52z\" (UniqueName: \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.212479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.212467 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.313577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.313496 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.313748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.313663 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.313748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.313721 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh52z\" (UniqueName: \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.313855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.313841 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.314431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.314407 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.316245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.316222 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.316419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.316400 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.331635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.331589 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh52z\" (UniqueName: \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\") pod \"maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.365640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.365585 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:46.566510 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.566392 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:46.580588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.580552 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:46.701322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:46.701291 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:21:46.703847 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:21:46.703814 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311c06ef_f28e_4539_9d73_0d0e51b80a2d.slice/crio-c98b7d74b23c18cce5533248e1d84d1b0ec73082cff0476a497621ea278bf5f1 WatchSource:0}: Error finding container c98b7d74b23c18cce5533248e1d84d1b0ec73082cff0476a497621ea278bf5f1: Status 404 returned error can't find the container with id c98b7d74b23c18cce5533248e1d84d1b0ec73082cff0476a497621ea278bf5f1 Oct 30 18:21:47.314953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:47.314905 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" event={"ID":"311c06ef-f28e-4539-9d73-0d0e51b80a2d","Type":"ContainerStarted","Data":"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1"} Oct 30 18:21:47.315437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:47.314967 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" event={"ID":"311c06ef-f28e-4539-9d73-0d0e51b80a2d","Type":"ContainerStarted","Data":"c98b7d74b23c18cce5533248e1d84d1b0ec73082cff0476a497621ea278bf5f1"} Oct 30 18:21:47.315437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:47.315069 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:21:47.334272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:47.334233 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:21:47.450628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:47.450521 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" podStartSLOduration=1.066533078 podStartE2EDuration="1.450499096s" totalImagesPullingTime="383.966018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:47.411920207 +0000 UTC m=+1258.232631986" watchObservedRunningTime="2026-10-30 18:21:47.450499096 +0000 UTC m=+1258.271210875" Oct 30 18:21:53.331144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:21:53.331113 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:22:10.460028 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:10.459988 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:22:12.759794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.759757 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7265-dfa54a"] Oct 30 18:22:12.764411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.764392 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:12.779874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.779845 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-dfa54a"] Oct 30 18:22:12.783128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.783095 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n7rzn\"" Oct 30 18:22:12.827528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.827493 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpgs\" (UniqueName: \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\") pod \"test-curl-7265-dfa54a\" (UID: \"74ec24fd-d2b6-4dd9-92a2-85f396120544\") " pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:12.929006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.928969 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpgs\" (UniqueName: \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\") pod \"test-curl-7265-dfa54a\" (UID: \"74ec24fd-d2b6-4dd9-92a2-85f396120544\") " pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:12.946185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:12.946152 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpgs\" (UniqueName: \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\") pod \"test-curl-7265-dfa54a\" (UID: \"74ec24fd-d2b6-4dd9-92a2-85f396120544\") " pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:13.079033 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:13.078944 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:13.384503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:13.384480 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-dfa54a"] Oct 30 18:22:13.386045 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:13.386010 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ec24fd_d2b6_4dd9_92a2_85f396120544.slice/crio-a20b7c9b2e95942d1182a891b991965a1a11288939193a313722fabc429acf63 WatchSource:0}: Error finding container a20b7c9b2e95942d1182a891b991965a1a11288939193a313722fabc429acf63: Status 404 returned error can't find the container with id a20b7c9b2e95942d1182a891b991965a1a11288939193a313722fabc429acf63 Oct 30 18:22:13.553032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:13.552991 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-dfa54a" event={"ID":"74ec24fd-d2b6-4dd9-92a2-85f396120544","Type":"ContainerStarted","Data":"4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651"} Oct 30 18:22:13.553205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:13.553045 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-dfa54a" event={"ID":"74ec24fd-d2b6-4dd9-92a2-85f396120544","Type":"ContainerStarted","Data":"a20b7c9b2e95942d1182a891b991965a1a11288939193a313722fabc429acf63"} Oct 30 18:22:13.588202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:13.588145 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7265-dfa54a" podStartSLOduration=1.588128614 podStartE2EDuration="1.588128614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:13.584711258 +0000 UTC m=+1284.405423034" watchObservedRunningTime="2026-10-30 18:22:13.588128614 +0000 UTC m=+1284.408840390" Oct 30 18:22:14.419427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.419388 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7265-dfa54a"] Oct 30 18:22:14.423776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.423747 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7265-dfa54a"] Oct 30 18:22:14.577463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.577426 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7265-cb4315"] Oct 30 18:22:14.578165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.578151 5024 cpu_manager.go:414] "containerMap: removing container" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" containerName="test-curl-7265-dfa54a" Oct 30 18:22:14.578219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.578166 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" containerName="test-curl-7265-dfa54a" Oct 30 18:22:14.578289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.578280 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" containerName="test-curl-7265-dfa54a" Oct 30 18:22:14.582233 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.582217 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:14.600215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.600179 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-cb4315"] Oct 30 18:22:14.657547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.657508 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dz5\" (UniqueName: \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\") pod \"test-curl-7265-cb4315\" (UID: \"f392b416-9aaa-4816-8cee-0ae7e37231e0\") " pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:14.758961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.758865 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dz5\" (UniqueName: \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\") pod \"test-curl-7265-cb4315\" (UID: \"f392b416-9aaa-4816-8cee-0ae7e37231e0\") " pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:14.775844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.775815 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dz5\" (UniqueName: \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\") pod \"test-curl-7265-cb4315\" (UID: \"f392b416-9aaa-4816-8cee-0ae7e37231e0\") " pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:14.894775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:14.894740 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:15.203462 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:15.203424 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-cb4315"] Oct 30 18:22:15.204967 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:15.204940 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf392b416_9aaa_4816_8cee_0ae7e37231e0.slice/crio-898b766de4b20ef27c0b7f1316c18c60eb28418442461ee8a8c6c7ae8850f44d WatchSource:0}: Error finding container 898b766de4b20ef27c0b7f1316c18c60eb28418442461ee8a8c6c7ae8850f44d: Status 404 returned error can't find the container with id 898b766de4b20ef27c0b7f1316c18c60eb28418442461ee8a8c6c7ae8850f44d Oct 30 18:22:15.569372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:15.569254 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-cb4315" event={"ID":"f392b416-9aaa-4816-8cee-0ae7e37231e0","Type":"ContainerStarted","Data":"118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa"} Oct 30 18:22:15.569372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:15.569310 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-cb4315" event={"ID":"f392b416-9aaa-4816-8cee-0ae7e37231e0","Type":"ContainerStarted","Data":"898b766de4b20ef27c0b7f1316c18c60eb28418442461ee8a8c6c7ae8850f44d"} Oct 30 18:22:15.570034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:15.569495 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7265-dfa54a" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" containerName="test-curl-7265-dfa54a" containerID="cri-o://4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651" gracePeriod=2 Oct 30 18:22:15.603892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:15.603819 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7265-cb4315" podStartSLOduration=1.603802505 podStartE2EDuration="1.603802505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:15.600460968 +0000 UTC m=+1286.421172744" watchObservedRunningTime="2026-10-30 18:22:15.603802505 +0000 UTC m=+1286.424514282" Oct 30 18:22:16.475943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.475911 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7265-cb4315"] Oct 30 18:22:16.485511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.485480 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7265-cb4315"] Oct 30 18:22:16.635982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.635946 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7265-a3d94b"] Oct 30 18:22:16.636696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.636673 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" containerName="test-curl-7265-cb4315" Oct 30 18:22:16.636696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.636689 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" containerName="test-curl-7265-cb4315" Oct 30 18:22:16.636821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.636789 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" containerName="test-curl-7265-cb4315" Oct 30 18:22:16.640207 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.640190 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:16.654963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.654930 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-a3d94b"] Oct 30 18:22:16.797171 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.797077 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct24c\" (UniqueName: \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\") pod \"test-curl-7265-a3d94b\" (UID: \"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa\") " pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:16.898707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.898670 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct24c\" (UniqueName: \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\") pod \"test-curl-7265-a3d94b\" (UID: \"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa\") " pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:16.916333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.916298 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct24c\" (UniqueName: \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\") pod \"test-curl-7265-a3d94b\" (UID: \"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa\") " pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:16.954319 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:16.954286 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:17.263700 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.263673 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-a3d94b"] Oct 30 18:22:17.267628 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:17.267567 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0caa3e_b93c_43e6_a6a8_c4984c485bfa.slice/crio-efca176e3b65226b4e88ad2efa6c35499b532f94895dee573d04972e42dc77f2 WatchSource:0}: Error finding container efca176e3b65226b4e88ad2efa6c35499b532f94895dee573d04972e42dc77f2: Status 404 returned error can't find the container with id efca176e3b65226b4e88ad2efa6c35499b532f94895dee573d04972e42dc77f2 Oct 30 18:22:17.587550 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.587516 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-a3d94b" event={"ID":"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa","Type":"ContainerStarted","Data":"c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3"} Oct 30 18:22:17.587725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.587570 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-a3d94b" event={"ID":"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa","Type":"ContainerStarted","Data":"efca176e3b65226b4e88ad2efa6c35499b532f94895dee573d04972e42dc77f2"} Oct 30 18:22:17.588579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.588544 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7265-cb4315" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" containerName="test-curl-7265-cb4315" containerID="cri-o://118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa" gracePeriod=2 Oct 30 18:22:17.620299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.620210 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7265-a3d94b" podStartSLOduration=1.6201949199999999 podStartE2EDuration="1.62019492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:17.619195491 +0000 UTC m=+1288.439907266" watchObservedRunningTime="2026-10-30 18:22:17.62019492 +0000 UTC m=+1288.440906695" Oct 30 18:22:17.741373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.741345 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:17.914590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.914556 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\" (UniqueName: \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\") pod \"74ec24fd-d2b6-4dd9-92a2-85f396120544\" (UID: \"74ec24fd-d2b6-4dd9-92a2-85f396120544\") " Oct 30 18:22:17.916986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:17.916949 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs" pod "74ec24fd-d2b6-4dd9-92a2-85f396120544" (UID: "74ec24fd-d2b6-4dd9-92a2-85f396120544"). InnerVolumeSpecName "kube-api-access-4fpgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:18.017085 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.017045 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpgs\" (UniqueName: \"kubernetes.io/projected/74ec24fd-d2b6-4dd9-92a2-85f396120544-kube-api-access-4fpgs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:18.472653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.472574 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7265-a3d94b"] Oct 30 18:22:18.479676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.478551 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7265-a3d94b"] Oct 30 18:22:18.596404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.596365 5024 generic.go:356] "Generic (PLEG): container finished" podID="74ec24fd-d2b6-4dd9-92a2-85f396120544" containerID="4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651" exitCode=137 Oct 30 18:22:18.596404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.596412 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:18.596711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.596461 5024 scope.go:122] "RemoveContainer" containerID="4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651" Oct 30 18:22:18.608515 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.608490 5024 scope.go:122] "RemoveContainer" containerID="4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651" Oct 30 18:22:18.608909 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:18.608887 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651\": container with ID starting with 4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651 not found: ID does not exist" containerID="4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651" Oct 30 18:22:18.608984 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.608918 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651"} err="failed to get container status \"4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651\": rpc error: code = NotFound desc = could not find container \"4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651\": container with ID starting with 4d8b017c54eb0a78bb99ccabf3d6557d49dba099c8fe0a0b60ccf45aa45fb651 not found: ID does not exist" Oct 30 18:22:18.618694 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:18.618663 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-dfa54a\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:18.628232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.628191 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7265-3647a9"] Oct 30 18:22:18.629003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.628986 5024 cpu_manager.go:414] "containerMap: removing container" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" containerName="test-curl-7265-a3d94b" Oct 30 18:22:18.629061 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.629005 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" containerName="test-curl-7265-a3d94b" Oct 30 18:22:18.629118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.629109 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" containerName="test-curl-7265-a3d94b" Oct 30 18:22:18.634028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.634009 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:18.644533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.644506 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbxd\" (UniqueName: \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\") pod \"test-curl-7265-3647a9\" (UID: \"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7\") " pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:18.647845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.647817 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-3647a9"] Oct 30 18:22:18.677792 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:18.677756 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-dfa54a\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:18.746673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.746544 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbxd\" (UniqueName: \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\") pod \"test-curl-7265-3647a9\" (UID: \"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7\") " pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:18.763832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.763804 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbxd\" (UniqueName: \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\") pod \"test-curl-7265-3647a9\" (UID: \"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7\") " pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:18.951975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:18.951940 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:19.253858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.253832 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7265-3647a9"] Oct 30 18:22:19.256248 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:19.256161 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98eaf9b0_9c38_4e47_ae65_e7a33a18f1e7.slice/crio-41141e8287060c99bccb63ac22315d35f4d840e3af3f04bc338a0690556fb152 WatchSource:0}: Error finding container 41141e8287060c99bccb63ac22315d35f4d840e3af3f04bc338a0690556fb152: Status 404 returned error can't find the container with id 41141e8287060c99bccb63ac22315d35f4d840e3af3f04bc338a0690556fb152 Oct 30 18:22:19.321246 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:19.321209 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-dfa54a\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" pod="opendatahub/test-curl-7265-dfa54a" Oct 30 18:22:19.322502 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.322480 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ec24fd-d2b6-4dd9-92a2-85f396120544" path="/var/lib/kubelet/pods/74ec24fd-d2b6-4dd9-92a2-85f396120544/volumes" Oct 30 18:22:19.605558 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.605522 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-3647a9" event={"ID":"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7","Type":"ContainerStarted","Data":"b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d"} Oct 30 18:22:19.605786 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.605577 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7265-3647a9" event={"ID":"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7","Type":"ContainerStarted","Data":"41141e8287060c99bccb63ac22315d35f4d840e3af3f04bc338a0690556fb152"} Oct 30 18:22:19.607414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.607389 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7265-a3d94b" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" containerName="test-curl-7265-a3d94b" containerID="cri-o://c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3" gracePeriod=2 Oct 30 18:22:19.640446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.640387 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7265-3647a9" podStartSLOduration=1.640371818 podStartE2EDuration="1.640371818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:19.636996695 +0000 UTC m=+1290.457708472" watchObservedRunningTime="2026-10-30 18:22:19.640371818 +0000 UTC m=+1290.461083594" Oct 30 18:22:19.757800 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.757774 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:19.862229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.862125 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\" (UniqueName: \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\") pod \"f392b416-9aaa-4816-8cee-0ae7e37231e0\" (UID: \"f392b416-9aaa-4816-8cee-0ae7e37231e0\") " Oct 30 18:22:19.864500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.864474 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5" pod "f392b416-9aaa-4816-8cee-0ae7e37231e0" (UID: "f392b416-9aaa-4816-8cee-0ae7e37231e0"). InnerVolumeSpecName "kube-api-access-j7dz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:19.964345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:19.964314 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dz5\" (UniqueName: \"kubernetes.io/projected/f392b416-9aaa-4816-8cee-0ae7e37231e0-kube-api-access-j7dz5\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:20.499789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.499755 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7265-3647a9"] Oct 30 18:22:20.503914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.503885 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7265-3647a9"] Oct 30 18:22:20.590229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.590186 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:22:20.591013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.590994 5024 cpu_manager.go:414] "containerMap: removing container" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" containerName="test-curl-7265-3647a9" Oct 30 18:22:20.591013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.591014 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" containerName="test-curl-7265-3647a9" Oct 30 18:22:20.591158 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.591142 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" containerName="test-curl-7265-3647a9" Oct 30 18:22:20.595444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.595424 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.628002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.627967 5024 generic.go:356] "Generic (PLEG): container finished" podID="f392b416-9aaa-4816-8cee-0ae7e37231e0" containerID="118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa" exitCode=137 Oct 30 18:22:20.628191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.628154 5024 scope.go:122] "RemoveContainer" containerID="118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa" Oct 30 18:22:20.628443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.628303 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:20.632351 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.632326 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:22:20.646114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.646092 5024 scope.go:122] "RemoveContainer" containerID="118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa" Oct 30 18:22:20.646444 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:20.646417 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa\": container with ID starting with 118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa not found: ID does not exist" containerID="118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa" Oct 30 18:22:20.646538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.646448 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa"} err="failed to get container status \"118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa\": rpc error: code = NotFound desc = could not find container \"118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa\": container with ID starting with 118ef676cefe46c5eef3a4c7b879159853d9eafc60792dd3082e8ece594489aa not found: ID does not exist" Oct 30 18:22:20.652462 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:20.652427 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-cb4315\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" pod="opendatahub/test-curl-7265-cb4315" Oct 30 18:22:20.675645 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.675591 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.675821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.675710 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.675936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.675918 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.675990 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.676051 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.676083 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.676101 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxkb\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676263 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.676243 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.676312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.676302 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.698224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.698193 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" Oct 30 18:22:20.778092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.777993 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778049 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778233 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778336 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778363 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778386 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxkb\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778884 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.778995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.778953 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.779050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.779009 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.779050 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.779033 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.779152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.779111 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.779295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.779270 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.779443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.779420 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.781046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.781023 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.781489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.781465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.796240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.796199 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.796358 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.796241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxkb\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:20.918394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:20.918359 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:21.248188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.248156 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:22:21.250187 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:21.250157 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8c7922_16a5_446e_b423_71341522105a.slice/crio-2b7a799cd5ce96e9ed9280236eb9ebcae3c2fae25d473555c7600c33d8d42bc6 WatchSource:0}: Error finding container 2b7a799cd5ce96e9ed9280236eb9ebcae3c2fae25d473555c7600c33d8d42bc6: Status 404 returned error can't find the container with id 2b7a799cd5ce96e9ed9280236eb9ebcae3c2fae25d473555c7600c33d8d42bc6 Oct 30 18:22:21.253544 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.253513 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:21.253634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.253582 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:21.253683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.253655 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:21.309001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.308969 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f392b416-9aaa-4816-8cee-0ae7e37231e0" path="/var/lib/kubelet/pods/f392b416-9aaa-4816-8cee-0ae7e37231e0/volumes" Oct 30 18:22:21.668848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.668766 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7265-3647a9" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" containerName="test-curl-7265-3647a9" containerID="cri-o://b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d" gracePeriod=2 Oct 30 18:22:21.670383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.670283 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" event={"ID":"5c8c7922-16a5-446e-b423-71341522105a","Type":"ContainerStarted","Data":"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631"} Oct 30 18:22:21.670383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.670348 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" event={"ID":"5c8c7922-16a5-446e-b423-71341522105a","Type":"ContainerStarted","Data":"2b7a799cd5ce96e9ed9280236eb9ebcae3c2fae25d473555c7600c33d8d42bc6"} Oct 30 18:22:21.719700 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.719629 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podStartSLOduration=1.7195864699999999 podStartE2EDuration="1.71958647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:21.713922523 +0000 UTC m=+1292.534634324" watchObservedRunningTime="2026-10-30 18:22:21.71958647 +0000 UTC m=+1292.540298249" Oct 30 18:22:21.981624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:21.980860 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:22.143869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.143836 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:22.245444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.245365 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\" (UniqueName: \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\") pod \"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa\" (UID: \"cd0caa3e-b93c-43e6-a6a8-c4984c485bfa\") " Oct 30 18:22:22.247914 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.247873 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c" pod "cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" (UID: "cd0caa3e-b93c-43e6-a6a8-c4984c485bfa"). InnerVolumeSpecName "kube-api-access-ct24c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:22.358624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.351366 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct24c\" (UniqueName: \"kubernetes.io/projected/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa-kube-api-access-ct24c\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:22.685477 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.685438 5024 generic.go:356] "Generic (PLEG): container finished" podID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" containerID="c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3" exitCode=137 Oct 30 18:22:22.685735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.685511 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:22.685735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.685532 5024 scope.go:122] "RemoveContainer" containerID="c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3" Oct 30 18:22:22.697357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.697334 5024 scope.go:122] "RemoveContainer" containerID="c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3" Oct 30 18:22:22.697754 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:22.697733 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3\": container with ID starting with c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3 not found: ID does not exist" containerID="c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3" Oct 30 18:22:22.697812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.697765 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3"} err="failed to get container status \"c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3\": rpc error: code = NotFound desc = could not find container \"c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3\": container with ID starting with c6ed13014fdba634824b137d50b48d1671bf50468e74f1e845d8637d9f4e57d3 not found: ID does not exist" Oct 30 18:22:22.708768 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:22.708721 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-a3d94b\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" pod="opendatahub/test-curl-7265-a3d94b" Oct 30 18:22:22.921590 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.921553 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:22.921834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:22.921672 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:23.308749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:23.308706 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0caa3e-b93c-43e6-a6a8-c4984c485bfa" path="/var/lib/kubelet/pods/cd0caa3e-b93c-43e6-a6a8-c4984c485bfa/volumes" Oct 30 18:22:23.919259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:23.918752 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:23.919259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:23.918839 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:23.996585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:23.996550 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:24.104268 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.104230 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\" (UniqueName: \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\") pod \"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7\" (UID: \"98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7\") " Oct 30 18:22:24.106805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.106772 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd" pod "98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" (UID: "98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7"). InnerVolumeSpecName "kube-api-access-8lbxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:24.207426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.207330 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lbxd\" (UniqueName: \"kubernetes.io/projected/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7-kube-api-access-8lbxd\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:24.707708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.707672 5024 generic.go:356] "Generic (PLEG): container finished" podID="760dff11-9ff5-4f46-a8e9-b34637ba2053" containerID="52569f0dac0805894e57902963bb1add7018cf43b868338ded3b70ab6b0dc06f" exitCode=0 Oct 30 18:22:24.708144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.707751 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" event={"ID":"760dff11-9ff5-4f46-a8e9-b34637ba2053","Type":"ContainerDied","Data":"52569f0dac0805894e57902963bb1add7018cf43b868338ded3b70ab6b0dc06f"} Oct 30 18:22:24.708980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.708959 5024 generic.go:356] "Generic (PLEG): container finished" podID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" containerID="b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d" exitCode=137 Oct 30 18:22:24.709090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.709004 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:24.709144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.709004 5024 scope.go:122] "RemoveContainer" containerID="b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d" Oct 30 18:22:24.720005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.719983 5024 scope.go:122] "RemoveContainer" containerID="b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d" Oct 30 18:22:24.720384 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:24.720364 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d\": container with ID starting with b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d not found: ID does not exist" containerID="b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d" Oct 30 18:22:24.720443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.720395 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d"} err="failed to get container status \"b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d\": rpc error: code = NotFound desc = could not find container \"b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d\": container with ID starting with b2de8deab51a8b69ea1f3e80fc49a703e5537d6a420693f6d1b89d86083fb92d not found: ID does not exist" Oct 30 18:22:24.785917 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:24.785869 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7265-3647a9\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" pod="opendatahub/test-curl-7265-3647a9" Oct 30 18:22:24.919410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.919375 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:24.919619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:24.919432 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:25.308118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:25.308079 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7" path="/var/lib/kubelet/pods/98eaf9b0-9c38-4e47-ae65-e7a33a18f1e7/volumes" Oct 30 18:22:25.919310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:25.919271 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:25.919858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:25.919336 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:25.990557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:25.990483 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:22:26.087357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.087310 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\" (UniqueName: \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\") pod \"760dff11-9ff5-4f46-a8e9-b34637ba2053\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " Oct 30 18:22:26.087674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.087650 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\") pod \"760dff11-9ff5-4f46-a8e9-b34637ba2053\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " Oct 30 18:22:26.087747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.087719 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\") pod \"760dff11-9ff5-4f46-a8e9-b34637ba2053\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " Oct 30 18:22:26.087836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.087820 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\") pod \"760dff11-9ff5-4f46-a8e9-b34637ba2053\" (UID: \"760dff11-9ff5-4f46-a8e9-b34637ba2053\") " Oct 30 18:22:26.088121 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088044 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle" pod "760dff11-9ff5-4f46-a8e9-b34637ba2053" (UID: "760dff11-9ff5-4f46-a8e9-b34637ba2053"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.088224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088115 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path" pod "760dff11-9ff5-4f46-a8e9-b34637ba2053" (UID: "760dff11-9ff5-4f46-a8e9-b34637ba2053"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:26.088429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088402 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle" pod "760dff11-9ff5-4f46-a8e9-b34637ba2053" (UID: "760dff11-9ff5-4f46-a8e9-b34637ba2053"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.088573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088557 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-trusted-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.088693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088577 5024 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/760dff11-9ff5-4f46-a8e9-b34637ba2053-archives-path\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.088693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.088587 5024 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/760dff11-9ff5-4f46-a8e9-b34637ba2053-service-ca-bundle\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.090032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.089997 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv" pod "760dff11-9ff5-4f46-a8e9-b34637ba2053" (UID: "760dff11-9ff5-4f46-a8e9-b34637ba2053"). InnerVolumeSpecName "kube-api-access-vf8mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.204192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.204149 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8mv\" (UniqueName: \"kubernetes.io/projected/760dff11-9ff5-4f46-a8e9-b34637ba2053-kube-api-access-vf8mv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.729804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.729776 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" Oct 30 18:22:26.729970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.729776 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t4rcs-7z9m4" event={"ID":"760dff11-9ff5-4f46-a8e9-b34637ba2053","Type":"ContainerDied","Data":"6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60"} Oct 30 18:22:26.729970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.729895 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd7fa24071d43d4f12be9d45b586db011a0fdeec7228b251e58715a42499d60" Oct 30 18:22:26.918991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.918954 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:26.919172 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.919020 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:26.979186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.979140 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:22:26.980270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.980190 5024 cpu_manager.go:414] "containerMap: removing container" podUID="760dff11-9ff5-4f46-a8e9-b34637ba2053" containerName="insights-gathering" Oct 30 18:22:26.980270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.980213 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760dff11-9ff5-4f46-a8e9-b34637ba2053" containerName="insights-gathering" Oct 30 18:22:26.980420 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.980368 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="760dff11-9ff5-4f46-a8e9-b34637ba2053" containerName="insights-gathering" Oct 30 18:22:26.984844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.984817 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:26.996127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:26.996089 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:22:27.024001 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.023958 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024183 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024109 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024218 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024267 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024317 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024349 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024472 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024424 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024580 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024555 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5zj\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.024647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.024623 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.071833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.071022 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" Oct 30 18:22:27.126146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126197 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126238 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126266 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126308 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5zj\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126564 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126358 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126563 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.126860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126825 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.127007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.126990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.127064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.127053 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.127154 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.127140 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.127341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.127321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.127843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.127824 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.128907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.128885 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.129173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.129158 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.146624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.146017 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.171625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.169347 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5zj\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.313803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.313712 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:27.682216 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.682176 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:22:27.687040 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:27.687011 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9ae01f_ab17_43a5_841d_d3e7c03e2d54.slice/crio-9d54320f0c40950b6a6ec880ca32c0b65f4687b9126cac1d7416c1eaaff7a4ea WatchSource:0}: Error finding container 9d54320f0c40950b6a6ec880ca32c0b65f4687b9126cac1d7416c1eaaff7a4ea: Status 404 returned error can't find the container with id 9d54320f0c40950b6a6ec880ca32c0b65f4687b9126cac1d7416c1eaaff7a4ea Oct 30 18:22:27.691176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.691131 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:27.691336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.691231 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:27.691336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.691275 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:27.743368 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.741776 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" event={"ID":"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54","Type":"ContainerStarted","Data":"9d54320f0c40950b6a6ec880ca32c0b65f4687b9126cac1d7416c1eaaff7a4ea"} Oct 30 18:22:27.918964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.918923 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 18:22:27.919204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:27.918987 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 18:22:28.779490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:28.779399 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" event={"ID":"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54","Type":"ContainerStarted","Data":"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f"} Oct 30 18:22:28.817688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:28.817564 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podStartSLOduration=2.817548536 podStartE2EDuration="2.817548536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:28.814790651 +0000 UTC m=+1299.635502419" watchObservedRunningTime="2026-10-30 18:22:28.817548536 +0000 UTC m=+1299.638260313" Oct 30 18:22:28.924547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:28.924513 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:28.924958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:28.924934 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:28.925726 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:28.925707 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:22:29.324894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:29.320739 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:29.325811 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:29.325779 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:22:29.325938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:29.325855 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:22:30.314849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:30.314810 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:22:30.315330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:30.314885 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:22:31.314260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:31.314229 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:22:31.314448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:31.314283 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:22:32.315074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:32.315033 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:22:32.315492 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:32.315098 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:22:33.314069 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:33.314033 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 18:22:33.314257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:33.314083 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 18:22:34.325286 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:34.325117 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:34.326897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:34.326873 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:34.329898 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:34.329855 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:22:35.657919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.657875 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:22:35.663426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.663397 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.686935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.686889 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:22:35.753828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.753788 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.753981 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754039 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754137 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754121 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754152 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754255 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754201 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754298 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkd7\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754364 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.754454 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.754441 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856143 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856108 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856158 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856211 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856244 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856267 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856324 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkd7\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856367 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856432 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856541 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856576 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.856679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856654 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.857021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.856995 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.857093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.857056 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.857161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.857134 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.859123 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.859101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.859481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.859440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.883531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.883494 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.883764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.883660 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkd7\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:35.982520 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:35.982428 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:36.340627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.340574 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:22:36.343004 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:22:36.342964 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8721759e_89db_4bf6_9ec1_58874188efb3.slice/crio-7a51971e9b927d0a1c2338eefb75bd6dd1695a97cbc57b85b071bf5ff24813f2 WatchSource:0}: Error finding container 7a51971e9b927d0a1c2338eefb75bd6dd1695a97cbc57b85b071bf5ff24813f2: Status 404 returned error can't find the container with id 7a51971e9b927d0a1c2338eefb75bd6dd1695a97cbc57b85b071bf5ff24813f2 Oct 30 18:22:36.346839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.346799 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:36.346911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.346893 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:36.346956 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.346937 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:22:36.866803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.866764 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerStarted","Data":"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41"} Oct 30 18:22:36.867272 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.866826 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerStarted","Data":"7a51971e9b927d0a1c2338eefb75bd6dd1695a97cbc57b85b071bf5ff24813f2"} Oct 30 18:22:36.921860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.921793 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podStartSLOduration=1.9217715480000002 podStartE2EDuration="1.921771548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:36.916038612 +0000 UTC m=+1307.736750388" watchObservedRunningTime="2026-10-30 18:22:36.921771548 +0000 UTC m=+1307.742483324" Oct 30 18:22:36.983572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:36.983538 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:37.998378 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:37.998343 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:38.900247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:38.900198 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:38.901488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:38.901461 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:22:40.464368 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:40.464323 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:22:41.436245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:41.436137 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb"] Oct 30 18:22:41.437670 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:41.437633 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" containerID="cri-o://a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738" gracePeriod=30 Oct 30 18:22:46.249088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.249048 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:22:46.249511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.249111 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:22:46.741346 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.741320 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:22:46.819661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819573 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.819836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819816 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.819910 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819882 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.819967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819927 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.819967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819956 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.820086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.819958 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.820086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820060 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.820197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820085 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.820197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820108 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.820197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820146 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.820367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820284 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.820367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820326 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\") pod \"3fa94489-c8bd-41ef-80de-2578bc7b000e\" (UID: \"3fa94489-c8bd-41ef-80de-2578bc7b000e\") " Oct 30 18:22:46.820705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820654 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.820705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.820681 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:46.821324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.821235 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa94489-c8bd-41ef-80de-2578bc7b000e-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.821324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.821261 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.821324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.821277 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.821324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.821290 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.821324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.821301 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.826624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.825120 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "kube-api-access-r9smd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:46.826624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.825355 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:46.826624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.825444 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:46.826624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.825526 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy" pod "3fa94489-c8bd-41ef-80de-2578bc7b000e" (UID: "3fa94489-c8bd-41ef-80de-2578bc7b000e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.923363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.923323 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9smd\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-kube-api-access-r9smd\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.923363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.923355 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.923363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.923368 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.923627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:46.923381 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa94489-c8bd-41ef-80de-2578bc7b000e-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.004832 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.004793 5024 generic.go:356] "Generic (PLEG): container finished" podID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerID="a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738" exitCode=0 Oct 30 18:22:47.005034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.004851 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" Oct 30 18:22:47.005034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.004869 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" event={"ID":"3fa94489-c8bd-41ef-80de-2578bc7b000e","Type":"ContainerDied","Data":"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738"} Oct 30 18:22:47.005034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.004949 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb" event={"ID":"3fa94489-c8bd-41ef-80de-2578bc7b000e","Type":"ContainerDied","Data":"1b8606e73f04fd2e8c09d8dd5c61606b3ab189c5757568a9343eb52409843c69"} Oct 30 18:22:47.005034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.004972 5024 scope.go:122] "RemoveContainer" containerID="a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738" Oct 30 18:22:47.025937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.025814 5024 scope.go:122] "RemoveContainer" containerID="a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738" Oct 30 18:22:47.031916 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:47.031884 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738\": container with ID starting with a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738 not found: ID does not exist" containerID="a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738" Oct 30 18:22:47.032053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:22:47.031924 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738"} err="failed to get container status \"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738\": rpc error: code = NotFound desc = could not find container \"a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738\": container with ID starting with a482749a25c450ff9ff1a4f35f4158ec42c05d667045e9ad70b40af18475a738 not found: ID does not exist" Oct 30 18:22:57.055280 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:22:57.055230 5024 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-143-94.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:07.055969 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:07.055922 5024 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-143-94.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:10.457565 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:10.457529 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:23:17.056528 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:17.056483 5024 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-143-94.ec2.internal?timeout=10s\": context deadline exceeded" Oct 30 18:23:27.057147 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:27.057101 5024 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-143-94.ec2.internal?timeout=10s\": context deadline exceeded" Oct 30 18:23:28.322126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322087 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fdf4da-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.322707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322155 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.322707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322173 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.322707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322206 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.322707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322245 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322713 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fp2dd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322759 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322773 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"simulator-w3-8b6cf1-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322371 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8sxnb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322376 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322368 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322405 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzpfj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322405 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322413 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322308 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322419 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322438 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8b6cf1-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322447 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322503 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-w0-6ce5eb-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322525 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322540 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322860 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322438 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ae3373-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322884 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"premium-w6-ef13cc-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322553 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322907 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322568 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322592 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322935 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322933 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322956 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322629 5024 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322690 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e5d168\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322818 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322280 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322961 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322546 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"distinct-2-w4-4ef6ac-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322673 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"distinct-w6-ef13cc-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322558 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.323008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322280 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j6bp8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322629 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323079 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323087 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323099 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323111 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323123 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323124 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323088 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323146 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323166 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e5d168-dockercfg-hlfd9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323188 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323211 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323241 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323255 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fh9d8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323287 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-njkgb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323324 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323381 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323424 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323443 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323438 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323480 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323488 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323459 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323488 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323532 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323540 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323562 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323571 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323580 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323614 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323643 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323659 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323674 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323680 5024 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323694 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323724 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323723 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323750 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323757 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323774 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323795 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323810 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323813 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323836 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323845 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323858 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323882 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323884 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"simulator-w5-fdf4da-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323909 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323923 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323948 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323965 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323990 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324029 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324030 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324036 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323850 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323913 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324067 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324073 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324091 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324097 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324113 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324129 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324135 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324142 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324154 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324176 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324202 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324214 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324224 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324239 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kkf44\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324179 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324259 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324281 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324283 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324289 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324303 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324310 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324321 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324321 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324337 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324358 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324369 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324377 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.323968 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324395 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324417 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324419 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324426 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324361 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wvmzn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324472 5024 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324483 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324488 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324496 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-dockercfg-d6sfq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324519 5024 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324529 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324536 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324556 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324585 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324589 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324629 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7f4dw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324651 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324678 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324699 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324723 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324746 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p4hk2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:28.324747 5024 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-143-94.ec2.internal?timeout=10s\": http2: client connection lost" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324769 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324773 5024 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324789 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324812 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324827 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324845 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324867 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324892 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"premium-w5-fdf4da-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324238 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.324810 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324258 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"default-dockercfg-52wg8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324922 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324948 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324969 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324114 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324997 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325019 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"distinct-w2-fde2c5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325039 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325060 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325092 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325103 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325125 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ff94j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325134 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325164 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325176 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325196 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325211 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325217 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325238 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cbhh9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325261 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ef6ac-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325265 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"premium-w1-fb2fa0-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325297 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325311 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-zw5t8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325324 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325344 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325346 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325366 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-fb2fa0-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325370 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325239 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325419 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325419 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325439 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325446 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-62n4h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325466 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kr5jk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325475 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325480 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-2-w5-fdf4da-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325300 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325505 5024 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325508 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qhhbf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325512 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ef13cc-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325525 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b2b596-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325536 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nqjgb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325543 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325558 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"e2e-lifecycle-model-fb0abbf7-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325561 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325576 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325633 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6wcdg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325589 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.324681 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325659 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325670 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325687 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325694 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325701 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325709 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325723 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325733 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325754 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325776 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325783 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325798 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nsbpz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325815 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg98\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325834 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v5bj5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325857 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325879 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325908 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325917 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325935 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325954 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"default-dockercfg-jl62g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325322 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325985 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8ljsd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325589 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325995 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:28.325860 5024 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/events\": http2: client connection lost" event=< Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: &Event{ObjectMeta:{e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb.18e361c2e0f1e11c openshift-ingress 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-ingress,Name:e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb,UID:3fa94489-c8bd-41ef-80de-2578bc7b000e,APIVersion:v1,ResourceVersion:25236,FieldPath:spec.containers{istio-proxy},},Reason:ProbeError,Message:Readiness probe error: HTTP probe failed with statuscode: 503 Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: body: Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: ,Source:EventSource{Component:kubelet,Host:ip-10-0-143-94.ec2.internal,},FirstTimestamp:2026-10-30 18:22:46.24908726 +0000 UTC m=+1317.069799033,LastTimestamp:2026-10-30 18:22:46.24908726 +0000 UTC m=+1317.069799033,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-94.ec2.internal,} Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: > Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325966 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.325756 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e5d168-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.326027 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.326026 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.326087 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:28.329500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:28.322719 5024 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-shsdq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:23:29.139900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.139853 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" Oct 30 18:23:29.162558 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.162516 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 18:23:29.163278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.163249 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"default-dockercfg-jl62g\"" Oct 30 18:23:29.163696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.163676 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 18:23:29.163791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.163706 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 18:23:29.163838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.163791 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.176692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.176646 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" Oct 30 18:23:29.177003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.176771 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" Oct 30 18:23:29.177003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.176968 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-62n4h\"" Oct 30 18:23:29.186960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.186910 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.188519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.188489 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:23:29.199162 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.199127 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:23:29.199421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.199128 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:23:29.199421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.199148 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.199973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.199947 5024 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:23:29.201282 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.201258 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.206273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.206132 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:23:29.217173 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.217136 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-fb2fa0-serving-cert\"" Oct 30 18:23:29.219869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.219839 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:23:29.223344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.223310 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:23:29.224405 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.224378 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.226025 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.226005 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:23:29.226516 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.226496 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.230356 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.230323 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 18:23:29.234167 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.234142 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"simulator-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:23:29.245090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.245049 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.247419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.247390 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.250349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.250325 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ff94j\"" Oct 30 18:23:29.254183 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.254161 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 18:23:29.255075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.255054 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 18:23:29.262169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.262141 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:23:29.268354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.268329 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:23:29.269286 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.269253 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:23:29.270512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.270491 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.270512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.270497 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wvmzn\"" Oct 30 18:23:29.273766 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.273743 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:23:29.274145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.274125 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.277747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.277725 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.281236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.281213 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.281382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.281282 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:23:29.283951 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.283932 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.290289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.290264 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:23:29.291968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.291927 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:23:29.292119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.291977 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nsbpz\"" Oct 30 18:23:29.298891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.298858 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.299049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.298970 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:23:29.299826 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.299804 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:23:29.308300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.307635 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.346548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.346520 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j6bp8\"" Oct 30 18:23:29.348114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.348082 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:23:29.349056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349030 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fh9d8\"" Oct 30 18:23:29.349179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349116 5024 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:23:29.349283 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349250 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" Oct 30 18:23:29.349394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349378 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:23:29.349503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349487 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.349649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349632 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 18:23:29.349753 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.348877 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.349829 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349676 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:23:29.349940 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.349922 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:23:29.350066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.350049 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:23:29.353313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.353287 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 18:23:29.359659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.357270 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:23:29.359906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.357370 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:23:29.362657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.362624 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 18:23:29.362885 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.362863 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.362976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.362866 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 18:23:29.363441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.363410 5024 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:23:29.366370 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.366322 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.366538 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.366449 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:23:29.370370 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.370343 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.371008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.370989 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.376354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.376326 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:23:29.381704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.380890 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e5d168\"" Oct 30 18:23:29.391944 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.391053 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 18:23:29.403741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.403515 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:23:29.403741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.403567 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:23:29.409200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.409167 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:23:29.409364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.409303 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.409523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.409505 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.414542 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.414515 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zjg98\"" Oct 30 18:23:29.415126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.415106 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" Oct 30 18:23:29.433217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.433187 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-zw5t8\"" Oct 30 18:23:29.433689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.433665 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" Oct 30 18:23:29.436818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.436785 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p4hk2\"" Oct 30 18:23:29.436986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.436845 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.437109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.437078 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"premium-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:23:29.437178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.437161 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" Oct 30 18:23:29.437262 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.437208 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fp2dd\"" Oct 30 18:23:29.437451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.437430 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.438277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.438143 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.452342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.452315 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" Oct 30 18:23:29.452512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.452376 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:23:29.452512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.452426 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 18:23:29.453487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.453470 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:23:29.459228 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.459206 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.464763 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.464737 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:23:29.469884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.469863 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.471209 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.471180 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:23:29.472146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.472128 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7f4dw\"" Oct 30 18:23:29.473755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.473731 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:23:29.475109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.475090 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:23:29.482447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.482421 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:23:29.482972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.482951 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-shsdq\"" Oct 30 18:23:29.483957 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.483938 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d6sfq\"" Oct 30 18:23:29.488560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.488535 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 18:23:29.488560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.488544 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.496791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.496766 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:23:29.497068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.497055 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.497133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.497088 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-njkgb\"" Oct 30 18:23:29.501481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.501457 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nqjgb\"" Oct 30 18:23:29.504251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.504224 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.504251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.504237 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.510293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.510249 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e5d168-dockercfg-hlfd9\"" Oct 30 18:23:29.510293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.510262 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.513260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.513233 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" Oct 30 18:23:29.518567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.518529 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 18:23:29.525012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.524988 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:23:29.526074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.526056 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"premium-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:23:29.527307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.527283 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 18:23:29.530241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.530216 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 18:23:29.532527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.532495 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:23:29.532527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.532499 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" Oct 30 18:23:29.536666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.536634 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:23:29.541544 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.541486 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:23:29.541544 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.541524 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.542264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.542247 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 18:23:29.543265 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.543249 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:23:29.547631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.547585 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.554005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.553978 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 18:23:29.556906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.556882 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:23:29.558954 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.558934 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 18:23:29.570030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.570003 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.570030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.570016 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8ljsd\"" Oct 30 18:23:29.577771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.577736 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v5bj5\"" Oct 30 18:23:29.581712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.581685 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:23:29.587932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.587903 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 18:23:29.587932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.587917 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 18:23:29.588719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.588703 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:23:29.588888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.588870 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.603202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.603170 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" Oct 30 18:23:29.607854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.607819 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.609945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.609923 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:23:29.611991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.611968 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"distinct-2-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:23:29.614010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.613990 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"premium-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:23:29.615947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.615918 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.619310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.619290 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cbhh9\"" Oct 30 18:23:29.623744 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.623720 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" Oct 30 18:23:29.624587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.624562 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:23:29.625696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.625678 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ae3373-serving-cert\"" Oct 30 18:23:29.630162 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.630138 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 18:23:29.630263 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.630148 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 18:23:29.630884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.630866 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-fdf4da-serving-cert\"" Oct 30 18:23:29.630973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.630918 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.644434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.644351 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ef13cc-serving-cert\"" Oct 30 18:23:29.665407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.665378 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.677275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.677237 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:23:29.679293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.679273 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.679770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.679744 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" Oct 30 18:23:29.681930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.681915 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.688759 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.688741 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.693736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.693716 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.703688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.703663 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:23:29.712444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.712421 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:23:29.712920 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.712901 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 18:23:29.715622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.715583 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8b6cf1-serving-cert\"" Oct 30 18:23:29.717759 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.717742 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 18:23:29.721210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.721190 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:23:29.723351 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.723334 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.724787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.724771 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.725665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.725648 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.726983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.726967 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:23:29.728486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.728465 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" Oct 30 18:23:29.730840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.730818 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8sxnb\"" Oct 30 18:23:29.731869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.731853 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"distinct-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:23:29.734194 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.734173 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 18:23:29.740302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.740282 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" Oct 30 18:23:29.742752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.742736 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.744766 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.744751 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.745809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.745795 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.751114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.751095 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.753152 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.753133 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.753233 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.753190 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:23:29.756301 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.756284 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b2b596-serving-cert\"" Oct 30 18:23:29.757476 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.757459 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzpfj\"" Oct 30 18:23:29.771464 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.771440 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"simulator-w3-8b6cf1-kserve-self-signed-certs\"" Oct 30 18:23:29.772438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.772422 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kkf44\"" Oct 30 18:23:29.773695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.773672 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.780512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.780481 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 18:23:29.786653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.786626 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.790616 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.790571 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" Oct 30 18:23:29.793786 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.793763 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:23:29.793904 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.793845 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:23:29.794549 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.794536 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 18:23:29.795785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.795770 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e5d168-serving-cert\"" Oct 30 18:23:29.797915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.797900 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.799363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.799348 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.799640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.799625 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:23:29.805732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.805707 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.808771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.808753 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.811135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.811117 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-qhhbf\"" Oct 30 18:23:29.814790 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.814759 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 18:23:29.823092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.823065 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 18:23:29.826553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.826531 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:23:29.832889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.832865 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:23:29.833005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.832894 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 18:23:29.834235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.834205 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.834345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.834302 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:23:29.840640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.840620 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" Oct 30 18:23:29.841630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.841610 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.842477 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.842456 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.844762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.844745 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-2-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:23:29.845869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.845851 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:23:29.852414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.852394 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:23:29.852525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.852458 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ef6ac-serving-cert\"" Oct 30 18:23:29.860818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.860788 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.870828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.870801 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.871918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.871899 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"distinct-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:23:29.875792 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.875745 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" Oct 30 18:23:29.880905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.880868 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" Oct 30 18:23:29.890087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.890057 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.891804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.891778 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"e2e-lifecycle-model-fb0abbf7-kserve-self-signed-certs\"" Oct 30 18:23:29.893673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.893652 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 18:23:29.899822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.899743 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.906208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.906180 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.906406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.906269 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 18:23:29.910799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.908428 5024 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:23:29.910799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.908740 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fb0abbf7\"/\"default-dockercfg-52wg8\"" Oct 30 18:23:29.910799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.910285 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:23:29.918621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.917719 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6wcdg\"" Oct 30 18:23:29.932065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.931979 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:23:29.932271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.932239 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.935710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.935686 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kr5jk\"" Oct 30 18:23:29.937480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.937455 5024 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:23:29.937666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.937547 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:23:29.945067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.945032 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xclgb"] Oct 30 18:23:29.945263 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.945092 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl","openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d","opendatahub/test-curl-7274-88f432","openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g","odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg"] Oct 30 18:23:29.946362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.946331 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" Oct 30 18:23:29.946362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.946363 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" Oct 30 18:23:29.946567 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.946553 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" containerName="istio-proxy" Oct 30 18:23:29.952012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.951980 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:29.957958 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.957913 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:29.964906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.964882 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:29.970693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.970662 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:29.978423 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.978315 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:29.981052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:29.981029 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n7rzn\"" Oct 30 18:23:30.060704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.060662 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.060853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.060744 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sqg\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.060853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.060779 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.060962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.060899 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fw5b\" (UniqueName: \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\") pod \"test-curl-7274-88f432\" (UID: \"5763aee8-9dcc-431b-b977-a0ef2f300f82\") " pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:30.060962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.060951 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.061037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061020 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061054 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.061129 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061097 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061168 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061201 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061217 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061249 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.061342 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061330 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhtv\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061360 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061405 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061433 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061464 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061487 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061562 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.061617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061587 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061648 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znnd\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061666 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.061734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061723 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061776 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061766 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061867 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061846 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.061903 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061875 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.061903 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061896 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.061972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061916 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.061972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.061947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.062039 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.062020 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.062071 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.062047 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.062102 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.062087 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv684\" (UniqueName: \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.079014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.078978 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9af5623f-serving-cert\"" Oct 30 18:23:30.164589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164490 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.164589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164564 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164653 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164677 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164709 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164732 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164755 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164828 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.164921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164910 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv684\" (UniqueName: \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.164972 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165002 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sqg\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165044 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165076 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fw5b\" (UniqueName: \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\") pod \"test-curl-7274-88f432\" (UID: \"5763aee8-9dcc-431b-b977-a0ef2f300f82\") " pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165123 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165153 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165189 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165258 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.165313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165299 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165335 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165362 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165419 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165483 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhtv\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165512 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165584 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165667 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.165772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165702 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.166185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165786 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.166185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165849 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.166185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165901 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znnd\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.166185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.165932 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.167022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.166982 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.167203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.167742 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.168216 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.168373 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.168463 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.168714 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.168687 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.169315 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.170026 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.170260 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.170296 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.170544 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.170845 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.171765 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.172980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.172938 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.180379 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.180348 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.191614 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.192243 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv684\" (UniqueName: \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.192681 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.192829 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fw5b\" (UniqueName: \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\") pod \"test-curl-7274-88f432\" (UID: \"5763aee8-9dcc-431b-b977-a0ef2f300f82\") " pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.192930 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sqg\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.194647 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znnd\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.196269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.196121 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.198878 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.196805 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.199469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.199446 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.199705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.199513 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.201232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.201188 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhtv\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.202454 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.202407 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.202803 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.202740 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.203130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.203104 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\") pod \"maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.204178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.204147 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.275978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.275946 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:30.288888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.288861 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:30.300989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.300961 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:30.314232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.314203 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:30.320973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.320932 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:30.765406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.765370 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g"] Oct 30 18:23:30.767659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.767538 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d"] Oct 30 18:23:30.818644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.796959 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.818644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.797077 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.818644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.797160 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.860498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.859841 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.860498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.859961 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:30.860498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:30.860019 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:31.153730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.153699 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg"] Oct 30 18:23:31.214623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.209734 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl"] Oct 30 18:23:31.247310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.241158 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:31.247310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.241531 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:31.247310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.241826 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:23:31.333749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.291036 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:23:31.516805 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.513292 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa94489-c8bd-41ef-80de-2578bc7b000e" path="/var/lib/kubelet/pods/3fa94489-c8bd-41ef-80de-2578bc7b000e/volumes" Oct 30 18:23:31.556640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.520822 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" event={"ID":"756952ee-14ed-44aa-adb5-96cba8be023c","Type":"ContainerStarted","Data":"25b5bdc95d8386674d87cc92cea84b42c3f7adf0268bd582392317b2462d43bd"} Oct 30 18:23:31.640679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.625818 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7274-88f432"] Oct 30 18:23:31.645270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.645231 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" event={"ID":"38aea5e9-5448-494c-b350-1867c4c56275","Type":"ContainerStarted","Data":"4580723a901538ddebb10d9c4935ff3ad0ba29afc5747f77d8fba9f1504f9b9f"} Oct 30 18:23:31.725699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.722945 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" event={"ID":"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1","Type":"ContainerStarted","Data":"2ae5a945091a0b91b783115afe180ee9b65e48d8532540e3624023c52be5162e"} Oct 30 18:23:31.741204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.741169 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" event={"ID":"ac42e75d-0afb-418b-b322-ca68dd3fdc26","Type":"ContainerStarted","Data":"5525555c1a8b52a6496120813da0ad064074551bc16185567030f6429a324b79"} Oct 30 18:23:31.743963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:31.743867 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7274-88f432" event={"ID":"5763aee8-9dcc-431b-b977-a0ef2f300f82","Type":"ContainerStarted","Data":"6b46d2b7f8242bd710fe5aac20390c0f4161521155be389854fc1498ed58f0bd"} Oct 30 18:23:32.772022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.770561 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" event={"ID":"38aea5e9-5448-494c-b350-1867c4c56275","Type":"ContainerStarted","Data":"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375"} Oct 30 18:23:32.776406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.775085 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" event={"ID":"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1","Type":"ContainerStarted","Data":"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627"} Oct 30 18:23:32.779247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.779188 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" event={"ID":"ac42e75d-0afb-418b-b322-ca68dd3fdc26","Type":"ContainerStarted","Data":"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6"} Oct 30 18:23:32.784957 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.784924 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7274-88f432" event={"ID":"5763aee8-9dcc-431b-b977-a0ef2f300f82","Type":"ContainerStarted","Data":"783d904f8dde0a123e6427d981b4b9e5e1328cc4c50759076d2ddd48eace1b24"} Oct 30 18:23:32.790480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.790445 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" event={"ID":"756952ee-14ed-44aa-adb5-96cba8be023c","Type":"ContainerStarted","Data":"4504475e7f38a204e16708c6df69dd9529818cd9f0fcc6540050475e4ac4159e"} Oct 30 18:23:32.791516 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.791490 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:32.875139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.875076 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podStartSLOduration=20.87505541 podStartE2EDuration="20.87505541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:32.849053636 +0000 UTC m=+1363.669765409" watchObservedRunningTime="2026-10-30 18:23:32.87505541 +0000 UTC m=+1363.695767189" Oct 30 18:23:32.930333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:32.930267 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podStartSLOduration=42.930244999 podStartE2EDuration="42.930244999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:32.925721027 +0000 UTC m=+1363.746432803" watchObservedRunningTime="2026-10-30 18:23:32.930244999 +0000 UTC m=+1363.750956776" Oct 30 18:23:33.026722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.021718 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7274-88f432" podStartSLOduration=10.021696354 podStartE2EDuration="10.021696354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:32.978555417 +0000 UTC m=+1363.799267194" watchObservedRunningTime="2026-10-30 18:23:33.021696354 +0000 UTC m=+1363.842408132" Oct 30 18:23:33.028115 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.028038 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" podStartSLOduration=4.386799544 podStartE2EDuration="5.028017466s" totalImagesPullingTime="641.217922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:33.019078019 +0000 UTC m=+1363.839789796" watchObservedRunningTime="2026-10-30 18:23:33.028017466 +0000 UTC m=+1363.848729243" Oct 30 18:23:33.091498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.091424 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podStartSLOduration=6.091398222 podStartE2EDuration="6.091398222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:33.067979456 +0000 UTC m=+1363.888691256" watchObservedRunningTime="2026-10-30 18:23:33.091398222 +0000 UTC m=+1363.912109999" Oct 30 18:23:33.280685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.278446 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:33.319624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.319052 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:33.319624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.319131 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:33.324559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.323485 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:33.324559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.323544 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:33.333628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.325143 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:33.333628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.325193 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:33.390624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.389678 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:33.390624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:33.389814 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:34.325393 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.323648 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:34.325393 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.323706 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:34.343935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.341708 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:34.343935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.341771 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:34.343935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.342017 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:34.343935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:34.342061 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:35.144825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.144787 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7274-88f432"] Oct 30 18:23:35.240007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.232495 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7274-88f432" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" containerName="test-curl-7274-88f432" containerID="cri-o://783d904f8dde0a123e6427d981b4b9e5e1328cc4c50759076d2ddd48eace1b24" gracePeriod=2 Oct 30 18:23:35.240007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.238467 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7274-88f432"] Oct 30 18:23:35.244722 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:35.244687 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7274-88f432\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:35.284625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.284068 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:35.284625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.284133 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:35.292625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.291012 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:35.292625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.291079 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:35.321624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.321055 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:35.321624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:35.321114 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:36.283649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.281437 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:36.283649 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.281514 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:36.336625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.307859 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:36.337277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.336920 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:36.337277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.337233 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:36.337482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:36.337302 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:37.300886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.289876 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:37.300886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.289942 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:37.300886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.290693 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:37.300886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.290733 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:37.328635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.317096 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:37.328635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.317156 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:37.441380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:37.441349 5024 generic.go:356] "Generic (PLEG): container finished" podID="5763aee8-9dcc-431b-b977-a0ef2f300f82" containerID="783d904f8dde0a123e6427d981b4b9e5e1328cc4c50759076d2ddd48eace1b24" exitCode=137 Oct 30 18:23:38.202632 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.201787 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:38.279624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.278997 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:23:38.279624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.279063 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 18:23:38.289627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.287693 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\" (UniqueName: \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\") pod \"5763aee8-9dcc-431b-b977-a0ef2f300f82\" (UID: \"5763aee8-9dcc-431b-b977-a0ef2f300f82\") " Oct 30 18:23:38.289627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.289350 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:23:38.289627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.289403 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 18:23:38.317682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.315718 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 18:23:38.317682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.315782 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 18:23:38.325210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.325170 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b" pod "5763aee8-9dcc-431b-b977-a0ef2f300f82" (UID: "5763aee8-9dcc-431b-b977-a0ef2f300f82"). InnerVolumeSpecName "kube-api-access-2fw5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:38.486398 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.456392 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fw5b\" (UniqueName: \"kubernetes.io/projected/5763aee8-9dcc-431b-b977-a0ef2f300f82-kube-api-access-2fw5b\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:23:38.528629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.528044 5024 scope.go:122] "RemoveContainer" containerID="783d904f8dde0a123e6427d981b4b9e5e1328cc4c50759076d2ddd48eace1b24" Oct 30 18:23:38.528629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:38.528276 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:38.657656 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:38.655300 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7274-88f432\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:39.298875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.298843 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:39.340347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.340313 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" path="/var/lib/kubelet/pods/5763aee8-9dcc-431b-b977-a0ef2f300f82/volumes" Oct 30 18:23:39.340792 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.340719 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:39.340839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.340804 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:39.340883 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.340839 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:23:39.342095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.342036 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:39.343269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.343247 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:23:39.367902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.367874 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:39.369670 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.369646 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:39.374495 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:39.374450 5024 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7274-88f432\" is forbidden: User \"system:node:ip-10-0-143-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-94.ec2.internal' and this object" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" pod="opendatahub/test-curl-7274-88f432" Oct 30 18:23:39.381907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:39.381877 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:23:40.238179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:40.238148 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:23:40.468057 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:23:40.468022 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:23:53.839668 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.839637 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:23:53.840383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.840369 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" containerName="test-curl-7274-88f432" Oct 30 18:23:53.840424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.840386 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" containerName="test-curl-7274-88f432" Oct 30 18:23:53.840523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.840514 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5763aee8-9dcc-431b-b977-a0ef2f300f82" containerName="test-curl-7274-88f432" Oct 30 18:23:53.845755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.845730 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:53.871138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.871035 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-295497-serving-cert\"" Oct 30 18:23:53.877916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.877879 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:23:53.912741 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.912701 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:53.912922 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.912824 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh5f\" (UniqueName: \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:53.912922 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.912864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:53.912994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:53.912946 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.014755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.014719 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.014971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.014957 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.015028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.015018 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh5f\" (UniqueName: \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.015070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.015042 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.015660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.015631 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.017613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.017580 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.017716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.017588 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.060687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.059985 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh5f\" (UniqueName: \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\") pod \"maas-api-e2e-api-a-295497-65d9c846cc-khbfp\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.162813 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.162560 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:54.627777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.626382 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:23:54.879806 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:54.879700 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" event={"ID":"bf4ddec7-9b80-404c-b783-a8d6f36f8074","Type":"ContainerStarted","Data":"4997a0362479203a8c4f0dc317840cb19254b2b9b921fe7b9c86fa2e63d32fc8"} Oct 30 18:23:55.889876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:55.889831 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" event={"ID":"bf4ddec7-9b80-404c-b783-a8d6f36f8074","Type":"ContainerStarted","Data":"296b17e01c60d2c472b55b4a52a1eb8f1fdeedf0184cca6070bc3aab22c19fb2"} Oct 30 18:23:55.890320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:55.889904 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:23:55.943985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:23:55.943885 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" podStartSLOduration=2.467386798 podStartE2EDuration="2.943861277s" totalImagesPullingTime="476.474479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:55.936045462 +0000 UTC m=+1386.756757233" watchObservedRunningTime="2026-10-30 18:23:55.943861277 +0000 UTC m=+1386.764573054" Oct 30 18:24:01.905706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:01.905621 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:24:10.460065 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:24:10.460025 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:24:10.586531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:10.586486 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:24:10.586833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:10.586798 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" containerID="cri-o://ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268" gracePeriod=30 Oct 30 18:24:10.631511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:10.631470 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:24:10.631802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:10.631749 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" containerID="cri-o://36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0" gracePeriod=30 Oct 30 18:24:15.948315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:15.948285 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:24:15.997554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:15.997526 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:24:16.032780 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.032753 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.032925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.032845 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.032925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.032907 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033044 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.032958 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033044 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.032988 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033146 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033186 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.033249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033193 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033354 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033309 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033386 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033394 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.033519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033428 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033451 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033488 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033513 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033546 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033611 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.033749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033631 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.033749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033647 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.033749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033723 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\") pod \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\" (UID: \"1a8ae026-a8a6-4524-9cab-767bcbec09a3\") " Oct 30 18:24:16.034015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.033806 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\") pod \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\" (UID: \"3fe5dff0-720b-42a4-b3c1-98943b0fc758\") " Oct 30 18:24:16.034453 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.034267 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.034963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.034807 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:16.035069 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.035003 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.035466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.035199 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.035466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.035274 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.035466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.035289 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.038876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.036216 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.039191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039116 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.039191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039147 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.039191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039164 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.039765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.036344 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "kube-api-access-l69nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:16.039765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.038700 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:16.039921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039896 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039928 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039942 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039954 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039969 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039985 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:16.040027 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.039998 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.040229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.040094 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:16.040229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.040108 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "kube-api-access-4w8bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:16.040229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.040149 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:16.041480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.041455 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token" pod "1a8ae026-a8a6-4524-9cab-767bcbec09a3" (UID: "1a8ae026-a8a6-4524-9cab-767bcbec09a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:16.041654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.041631 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy" pod "3fe5dff0-720b-42a4-b3c1-98943b0fc758" (UID: "3fe5dff0-720b-42a4-b3c1-98943b0fc758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141616 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141646 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141656 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141664 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe5dff0-720b-42a4-b3c1-98943b0fc758-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141674 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141681 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141689 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8bj\" (UniqueName: \"kubernetes.io/projected/1a8ae026-a8a6-4524-9cab-767bcbec09a3-kube-api-access-4w8bj\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141700 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8ae026-a8a6-4524-9cab-767bcbec09a3-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.141704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.141710 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l69nd\" (UniqueName: \"kubernetes.io/projected/3fe5dff0-720b-42a4-b3c1-98943b0fc758-kube-api-access-l69nd\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:16.193687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.193653 5024 generic.go:356] "Generic (PLEG): container finished" podID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerID="ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268" exitCode=0 Oct 30 18:24:16.193850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.193702 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" Oct 30 18:24:16.193850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.193737 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" event={"ID":"3fe5dff0-720b-42a4-b3c1-98943b0fc758","Type":"ContainerDied","Data":"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268"} Oct 30 18:24:16.193850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.193794 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd" event={"ID":"3fe5dff0-720b-42a4-b3c1-98943b0fc758","Type":"ContainerDied","Data":"8f5e0cde5afdb57c9b1c51844366b1d5868a5a955ee4f73057cb20661a12c4c0"} Oct 30 18:24:16.193850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.193828 5024 scope.go:122] "RemoveContainer" containerID="ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268" Oct 30 18:24:16.195246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.195225 5024 generic.go:356] "Generic (PLEG): container finished" podID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerID="36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0" exitCode=0 Oct 30 18:24:16.195376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.195279 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" Oct 30 18:24:16.195376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.195306 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" event={"ID":"1a8ae026-a8a6-4524-9cab-767bcbec09a3","Type":"ContainerDied","Data":"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0"} Oct 30 18:24:16.195376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.195361 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5" event={"ID":"1a8ae026-a8a6-4524-9cab-767bcbec09a3","Type":"ContainerDied","Data":"9a513ceb535b099963807fef9ba592605f5ec6c89b5e10a2d51f787ea69c1896"} Oct 30 18:24:16.205795 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.205775 5024 scope.go:122] "RemoveContainer" containerID="ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268" Oct 30 18:24:16.206134 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:24:16.206105 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268\": container with ID starting with ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268 not found: ID does not exist" containerID="ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268" Oct 30 18:24:16.206244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.206147 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268"} err="failed to get container status \"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268\": rpc error: code = NotFound desc = could not find container \"ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268\": container with ID starting with ea2ecc91279ee930d89842d82d39386b6425680deaee4ee4c26932a6c010f268 not found: ID does not exist" Oct 30 18:24:16.206244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.206171 5024 scope.go:122] "RemoveContainer" containerID="36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0" Oct 30 18:24:16.230292 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.230270 5024 scope.go:122] "RemoveContainer" containerID="36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0" Oct 30 18:24:16.230703 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:24:16.230667 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0\": container with ID starting with 36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0 not found: ID does not exist" containerID="36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0" Oct 30 18:24:16.230825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.230710 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0"} err="failed to get container status \"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0\": rpc error: code = NotFound desc = could not find container \"36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0\": container with ID starting with 36386d118db217c03161a7a416ca93c6385e3432bdaf075b278a087f170e13f0 not found: ID does not exist" Oct 30 18:24:16.252039 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.252006 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:24:16.258584 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.258554 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-r7nl5"] Oct 30 18:24:16.289587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.289553 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:24:16.295306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.295284 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-8tghd"] Oct 30 18:24:16.419115 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419034 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:24:16.419781 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419766 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" Oct 30 18:24:16.419833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419782 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" Oct 30 18:24:16.419833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419828 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" Oct 30 18:24:16.419901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419833 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" Oct 30 18:24:16.419950 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419940 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" containerName="istio-proxy" Oct 30 18:24:16.419983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.419955 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" containerName="istio-proxy" Oct 30 18:24:16.425473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.425451 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.441147 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.440486 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9066aa-serving-cert\"" Oct 30 18:24:16.447012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.446982 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:24:16.548459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.548428 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.548673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.548478 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.548673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.548644 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.548894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.548863 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mnb\" (UniqueName: \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.653196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.653148 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mnb\" (UniqueName: \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.653397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.653361 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.653502 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.653410 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.653554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.653535 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.654405 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.654373 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.661303 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.661262 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.665228 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.665205 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.674327 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.674250 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mnb\" (UniqueName: \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\") pod \"maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:16.746299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:16.746262 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:17.065056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.065017 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:24:17.205995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.205949 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" event={"ID":"c1db325a-4c0b-4381-8a92-16eda65f4579","Type":"ContainerStarted","Data":"e4fefc22e003221830a2ffdd46538606dcbf2c51a9dd610bd1ca018befc82fdf"} Oct 30 18:24:17.326698 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.326574 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8ae026-a8a6-4524-9cab-767bcbec09a3" path="/var/lib/kubelet/pods/1a8ae026-a8a6-4524-9cab-767bcbec09a3/volumes" Oct 30 18:24:17.327448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.327426 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe5dff0-720b-42a4-b3c1-98943b0fc758" path="/var/lib/kubelet/pods/3fe5dff0-720b-42a4-b3c1-98943b0fc758/volumes" Oct 30 18:24:17.715274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.715231 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:24:17.720266 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.720242 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.737974 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738080 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738133 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738174 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738205 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738239 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738265 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxgc\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738305 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.742794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.738347 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.781623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.746732 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9af5623f-openshift-default-dockercfg-vxdf5\"" Oct 30 18:24:17.832244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.832206 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:24:17.839802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.839768 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.839802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.839806 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxgc\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840061 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.839845 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840061 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.839886 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840160 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840216 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840262 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840291 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840422 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840504 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840709 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.840943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.840907 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.841141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.841121 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.843463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.843440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.843563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.843517 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.858838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.858812 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:17.858987 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:17.858966 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxgc\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:18.125255 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.125163 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:18.221026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.220640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" event={"ID":"c1db325a-4c0b-4381-8a92-16eda65f4579","Type":"ContainerStarted","Data":"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6"} Oct 30 18:24:18.221026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.220973 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:18.266345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.266280 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" podStartSLOduration=1.972668506 podStartE2EDuration="2.266259742s" totalImagesPullingTime="293.591236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:18.257196391 +0000 UTC m=+1409.077908168" watchObservedRunningTime="2026-10-30 18:24:18.266259742 +0000 UTC m=+1409.086971519" Oct 30 18:24:18.457472 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.457434 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:24:18.457701 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:24:18.457669 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabfa326_cef6_471b_b332_686a2827c2c8.slice/crio-3df79bfe269580253f1b6bcc1603d043bf3e4574e78d14beb9bac21a28dad3d3 WatchSource:0}: Error finding container 3df79bfe269580253f1b6bcc1603d043bf3e4574e78d14beb9bac21a28dad3d3: Status 404 returned error can't find the container with id 3df79bfe269580253f1b6bcc1603d043bf3e4574e78d14beb9bac21a28dad3d3 Oct 30 18:24:18.461081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.461043 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:18.461202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.461118 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:18.461202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:18.461147 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:19.229860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:19.229817 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerStarted","Data":"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769"} Oct 30 18:24:19.229860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:19.229877 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerStarted","Data":"3df79bfe269580253f1b6bcc1603d043bf3e4574e78d14beb9bac21a28dad3d3"} Oct 30 18:24:19.283261 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:19.283189 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podStartSLOduration=2.283169512 podStartE2EDuration="2.283169512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:19.280442342 +0000 UTC m=+1410.101154118" watchObservedRunningTime="2026-10-30 18:24:19.283169512 +0000 UTC m=+1410.103881289" Oct 30 18:24:20.125522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:20.125478 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:20.129488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:20.127927 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:20.129488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:20.127998 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:21.126013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:21.125977 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:21.126485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:21.126034 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:22.125956 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:22.125922 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:22.126112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:22.125977 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:23.126102 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:23.126069 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:23.126535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:23.126136 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:24.126226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:24.126188 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:24.126720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:24.126246 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:24.236214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:24.236180 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:24:25.126427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:25.126393 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:25.126935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:25.126474 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:25.611362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:25.611321 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:24:25.611996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:25.611943 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" containerID="cri-o://317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f" gracePeriod=30 Oct 30 18:24:26.126358 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.126320 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:26.126535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.126377 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:26.651416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.651377 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:24:26.657321 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.657296 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.674736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.674706 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:24:26.702181 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.702142 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-18cd5a-serving-cert\"" Oct 30 18:24:26.805223 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.805183 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.805442 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.805326 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.805442 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.805371 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.805572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.805560 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgfd\" (UniqueName: \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.907473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.907385 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgfd\" (UniqueName: \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.907711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.907557 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.907711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.907621 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.907711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.907652 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.908284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.908263 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.910335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.910313 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.910433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.910369 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.925165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.925140 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgfd\" (UniqueName: \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\") pod \"maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:26.979006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:26.977048 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:27.128623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:27.128414 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:27.128623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:27.128491 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:27.324342 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:24:27.324312 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6861c2bd_8e3d_4e6d_98dc_5a05804aca6d.slice/crio-30a91f22719a0770071fa660da2e3ffee7b219edc8c92caa3ebc9fbe9a89a064 WatchSource:0}: Error finding container 30a91f22719a0770071fa660da2e3ffee7b219edc8c92caa3ebc9fbe9a89a064: Status 404 returned error can't find the container with id 30a91f22719a0770071fa660da2e3ffee7b219edc8c92caa3ebc9fbe9a89a064 Oct 30 18:24:27.324639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:27.324584 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:24:28.126057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.125961 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:28.126057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.126018 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:28.311660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.311266 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" event={"ID":"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d","Type":"ContainerStarted","Data":"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b"} Oct 30 18:24:28.311660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.311334 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" event={"ID":"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d","Type":"ContainerStarted","Data":"30a91f22719a0770071fa660da2e3ffee7b219edc8c92caa3ebc9fbe9a89a064"} Oct 30 18:24:28.311660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.311358 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:28.351133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:28.351080 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" podStartSLOduration=1.9471082910000002 podStartE2EDuration="2.351065501s" totalImagesPullingTime="403.95721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:28.346851038 +0000 UTC m=+1419.167562816" watchObservedRunningTime="2026-10-30 18:24:28.351065501 +0000 UTC m=+1419.171777276" Oct 30 18:24:29.125975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:29.125938 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:29.126248 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:29.126004 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:30.125734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:30.125702 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:30.126141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:30.125755 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:30.935159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:30.935127 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:24:31.035410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035363 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.035647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035555 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.035750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035645 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.035815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035801 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.035935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035908 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.036013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035927 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.036013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035950 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.036013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.035996 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.036710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.036034 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.036710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.036068 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\") pod \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\" (UID: \"552191e9-9850-42fc-8fc3-1bfa5c05bca5\") " Oct 30 18:24:31.036891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.036818 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.037040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.037018 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.037119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.037046 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.037119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.037072 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.037119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.037087 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.037309 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.037284 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:31.038530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.038503 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.039399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.039371 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:31.039811 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.039669 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "kube-api-access-zlxxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.039811 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.039778 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy" pod "552191e9-9850-42fc-8fc3-1bfa5c05bca5" (UID: "552191e9-9850-42fc-8fc3-1bfa5c05bca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.126476 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.126442 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:31.127024 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.126504 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:31.138783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138746 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138781 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138797 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138809 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxxj\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-kube-api-access-zlxxj\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138821 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138835 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552191e9-9850-42fc-8fc3-1bfa5c05bca5-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.138966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.138847 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552191e9-9850-42fc-8fc3-1bfa5c05bca5-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.340266 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.340236 5024 generic.go:356] "Generic (PLEG): container finished" podID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerID="317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f" exitCode=0 Oct 30 18:24:31.340404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.340284 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" event={"ID":"552191e9-9850-42fc-8fc3-1bfa5c05bca5","Type":"ContainerDied","Data":"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f"} Oct 30 18:24:31.340404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.340287 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" Oct 30 18:24:31.340404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.340324 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd" event={"ID":"552191e9-9850-42fc-8fc3-1bfa5c05bca5","Type":"ContainerDied","Data":"759eb0e871e55c1192c82cbab58ef607c0704dd85ce4a8b4bebfe3be5799c8df"} Oct 30 18:24:31.340404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.340345 5024 scope.go:122] "RemoveContainer" containerID="317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f" Oct 30 18:24:31.353954 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.353873 5024 scope.go:122] "RemoveContainer" containerID="317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f" Oct 30 18:24:31.354384 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:24:31.354358 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f\": container with ID starting with 317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f not found: ID does not exist" containerID="317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f" Oct 30 18:24:31.354523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.354394 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f"} err="failed to get container status \"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f\": rpc error: code = NotFound desc = could not find container \"317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f\": container with ID starting with 317ca371701fa115e576da86628c7af5ceb8b473650459f88cb61a98daf59f8f not found: ID does not exist" Oct 30 18:24:31.400937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.400905 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:24:31.411114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:31.411077 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rqwtd"] Oct 30 18:24:32.125719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.125681 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:32.125919 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.125752 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:32.764992 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.764958 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:24:32.765900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.765879 5024 cpu_manager.go:414] "containerMap: removing container" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" Oct 30 18:24:32.765900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.765901 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" Oct 30 18:24:32.766022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.766013 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" containerName="istio-proxy" Oct 30 18:24:32.771402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.771371 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.792179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.792146 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:24:32.870792 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.870745 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.870990 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871188 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871235 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871367 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871344 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7674g\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871384 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871500 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871546 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.871671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.871576 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.973637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.973582 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.973637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.973648 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.973912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.973678 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.973912 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.973745 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974017 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.973907 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974129 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974111 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974122 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974160 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974167 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974216 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974238 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7674g\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974273 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974388 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.974710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.974688 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.976306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.976284 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.976504 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.976488 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.992684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.992652 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7674g\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:32.992820 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:32.992804 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:33.086259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.086170 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:33.125364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.125334 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:33.125524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.125501 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.125588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.125569 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:33.308505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.308471 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552191e9-9850-42fc-8fc3-1bfa5c05bca5" path="/var/lib/kubelet/pods/552191e9-9850-42fc-8fc3-1bfa5c05bca5/volumes" Oct 30 18:24:33.418710 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:24:33.418681 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16e7315_0881_41cc_898d_378727f3335d.slice/crio-fc11bc397e50f19b6c399a0c58a2b9cc09e30b1fc462e5e3daa10cc2a7b0fb07 WatchSource:0}: Error finding container fc11bc397e50f19b6c399a0c58a2b9cc09e30b1fc462e5e3daa10cc2a7b0fb07: Status 404 returned error can't find the container with id fc11bc397e50f19b6c399a0c58a2b9cc09e30b1fc462e5e3daa10cc2a7b0fb07 Oct 30 18:24:33.418994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.418970 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:24:33.421736 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.421704 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:33.421837 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.421775 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:33.421837 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:33.421802 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:24:34.136335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.136297 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.136985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.136952 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:34.326217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.326188 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:24:34.369917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.369879 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerStarted","Data":"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92"} Oct 30 18:24:34.370127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.369950 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerStarted","Data":"fc11bc397e50f19b6c399a0c58a2b9cc09e30b1fc462e5e3daa10cc2a7b0fb07"} Oct 30 18:24:34.423939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:34.423868 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podStartSLOduration=2.42384504 podStartE2EDuration="2.42384504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:34.416406213 +0000 UTC m=+1425.237117989" watchObservedRunningTime="2026-10-30 18:24:34.42384504 +0000 UTC m=+1425.244556817" Oct 30 18:24:35.086670 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:35.086567 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:35.088133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:35.088110 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.088195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:35.088157 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 18:24:35.126290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:35.126253 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.126484 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:35.126304 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:36.086775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:36.086727 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.087194 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:36.086792 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 18:24:36.125775 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:36.125732 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.125924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:36.125799 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:37.086942 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:37.086912 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.087433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:37.086974 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 18:24:37.126162 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:37.126117 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.126335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:37.126191 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:24:38.086737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:38.086697 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 18:24:38.086934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:38.086755 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 18:24:38.130783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:38.130745 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:38.131809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:38.131789 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:24:39.113002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:39.112959 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:39.128710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:39.128291 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:39.146406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:39.146376 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:24:40.475193 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:24:40.475153 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:24:49.947789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:49.947692 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:24:49.971365 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:49.971326 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:24:50.032634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:24:50.032581 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:25:04.972008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:04.971966 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" containerID="cri-o://9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1" gracePeriod=45 Oct 30 18:25:04.994160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:04.994119 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" containerID="cri-o://3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8" gracePeriod=45 Oct 30 18:25:05.062122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.062074 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" containerID="cri-o://c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8" gracePeriod=45 Oct 30 18:25:05.140626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.140550 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Oct 30 18:25:05.228189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.228091 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Oct 30 18:25:05.305407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.305375 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:25:05.406411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.406366 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:25:05.447978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.447944 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:25:05.467923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.467887 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.467948 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.467989 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\" (UniqueName: \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468019 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468051 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468082 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468109 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468238 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\" (UniqueName: \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468271 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468316 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.468450 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468366 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468455 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468572 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\" (UniqueName: \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\") pod \"d367b5f9-6441-46e7-968c-11034679ef97\" (UID: \"d367b5f9-6441-46e7-968c-11034679ef97\") " Oct 30 18:25:05.468737 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468631 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468767 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\" (UniqueName: \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.468905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468841 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\") pod \"b42b2b39-1e15-41c7-82ae-d902239de481\" (UID: \"b42b2b39-1e15-41c7-82ae-d902239de481\") " Oct 30 18:25:05.468905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468878 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.469153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.468974 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\") pod \"5bff7d2b-a767-434f-83c8-aa847746f21c\" (UID: \"5bff7d2b-a767-434f-83c8-aa847746f21c\") " Oct 30 18:25:05.470539 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.470507 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.470844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.470817 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.471088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.471061 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.471163 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.471081 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.471361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.471340 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.471574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.471553 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.472704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.472664 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.473159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.472850 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.475417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.475391 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.475727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.475703 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.479056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.478989 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:05.479164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.479143 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.480808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.480784 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.482141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.482116 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:05.482244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.482138 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm" pod "5bff7d2b-a767-434f-83c8-aa847746f21c" (UID: "5bff7d2b-a767-434f-83c8-aa847746f21c"). InnerVolumeSpecName "kube-api-access-tjdwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:05.482244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.482188 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "kube-api-access-5f5tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:05.482482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.482238 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh" pod "d367b5f9-6441-46e7-968c-11034679ef97" (UID: "d367b5f9-6441-46e7-968c-11034679ef97"). InnerVolumeSpecName "kube-api-access-qvlzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:05.484323 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.484295 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs" pod "b42b2b39-1e15-41c7-82ae-d902239de481" (UID: "b42b2b39-1e15-41c7-82ae-d902239de481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:05.572037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572001 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572038 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572050 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d367b5f9-6441-46e7-968c-11034679ef97-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572063 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572075 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdwm\" (UniqueName: \"kubernetes.io/projected/5bff7d2b-a767-434f-83c8-aa847746f21c-kube-api-access-tjdwm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572088 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572100 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572111 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572122 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572132 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572145 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f5tz\" (UniqueName: \"kubernetes.io/projected/b42b2b39-1e15-41c7-82ae-d902239de481-kube-api-access-5f5tz\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572156 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572170 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bff7d2b-a767-434f-83c8-aa847746f21c-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572181 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b42b2b39-1e15-41c7-82ae-d902239de481-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572193 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42b2b39-1e15-41c7-82ae-d902239de481-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572204 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlzh\" (UniqueName: \"kubernetes.io/projected/d367b5f9-6441-46e7-968c-11034679ef97-kube-api-access-qvlzh\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572215 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d367b5f9-6441-46e7-968c-11034679ef97-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.572253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.572227 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bff7d2b-a767-434f-83c8-aa847746f21c-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.807949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.807836 5024 generic.go:356] "Generic (PLEG): container finished" podID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerID="c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8" exitCode=0 Oct 30 18:25:05.808141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.807950 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" event={"ID":"5bff7d2b-a767-434f-83c8-aa847746f21c","Type":"ContainerDied","Data":"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8"} Oct 30 18:25:05.808141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.807992 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" event={"ID":"5bff7d2b-a767-434f-83c8-aa847746f21c","Type":"ContainerDied","Data":"9d9738b208133a9e18f341ccb291ab828c2fb77ce8135eb2b9d27dce5d7e8b08"} Oct 30 18:25:05.808141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.808015 5024 scope.go:122] "RemoveContainer" containerID="c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8" Oct 30 18:25:05.808336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.808179 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m" Oct 30 18:25:05.821191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.821158 5024 generic.go:356] "Generic (PLEG): container finished" podID="b42b2b39-1e15-41c7-82ae-d902239de481" containerID="9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1" exitCode=0 Oct 30 18:25:05.821345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.821271 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" event={"ID":"b42b2b39-1e15-41c7-82ae-d902239de481","Type":"ContainerDied","Data":"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1"} Oct 30 18:25:05.821345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.821309 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" event={"ID":"b42b2b39-1e15-41c7-82ae-d902239de481","Type":"ContainerDied","Data":"e273bc69b7f31c83870f8780c8f2bbd238329f848b74e2139a029e2525a7554d"} Oct 30 18:25:05.821469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.821386 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26" Oct 30 18:25:05.826164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.826137 5024 generic.go:356] "Generic (PLEG): container finished" podID="d367b5f9-6441-46e7-968c-11034679ef97" containerID="3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8" exitCode=0 Oct 30 18:25:05.826335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.826312 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" event={"ID":"d367b5f9-6441-46e7-968c-11034679ef97","Type":"ContainerDied","Data":"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8"} Oct 30 18:25:05.826465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.826354 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" event={"ID":"d367b5f9-6441-46e7-968c-11034679ef97","Type":"ContainerDied","Data":"291c9ab400a6e5024bcda2146ae652173fdf1fe58b2ffadedf3915b6e1538dd1"} Oct 30 18:25:05.826465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.826420 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv" Oct 30 18:25:05.839459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.839353 5024 scope.go:122] "RemoveContainer" containerID="c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8" Oct 30 18:25:05.839877 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:05.839841 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8\": container with ID starting with c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8 not found: ID does not exist" containerID="c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8" Oct 30 18:25:05.839986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.839888 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8"} err="failed to get container status \"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8\": rpc error: code = NotFound desc = could not find container \"c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8\": container with ID starting with c9026557e4d0529f6e3aee300c1a37298d9d386344c5542374b63492e1935df8 not found: ID does not exist" Oct 30 18:25:05.839986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.839911 5024 scope.go:122] "RemoveContainer" containerID="9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1" Oct 30 18:25:05.851481 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.851443 5024 scope.go:122] "RemoveContainer" containerID="9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1" Oct 30 18:25:05.851835 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:05.851811 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1\": container with ID starting with 9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1 not found: ID does not exist" containerID="9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1" Oct 30 18:25:05.851924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.851842 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1"} err="failed to get container status \"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1\": rpc error: code = NotFound desc = could not find container \"9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1\": container with ID starting with 9f54bcaf810837aa6ff6e3a91348e10c9a12b42ae3c06b8a4ba8fb4c01f245c1 not found: ID does not exist" Oct 30 18:25:05.851924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.851862 5024 scope.go:122] "RemoveContainer" containerID="3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8" Oct 30 18:25:05.867356 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.867336 5024 scope.go:122] "RemoveContainer" containerID="3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8" Oct 30 18:25:05.867687 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:05.867658 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8\": container with ID starting with 3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8 not found: ID does not exist" containerID="3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8" Oct 30 18:25:05.867801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.867695 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8"} err="failed to get container status \"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8\": rpc error: code = NotFound desc = could not find container \"3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8\": container with ID starting with 3676573fcc0626bed8c3ad17fa4e407c1c5ab35db97746cb8ec71e10d552a1d8 not found: ID does not exist" Oct 30 18:25:05.896507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.896473 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:25:05.904801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.904773 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/simulator-w5-fdf4da-kserve-9996bf9df-6w75m"] Oct 30 18:25:05.933770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.932433 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:25:05.941755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.941717 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-2-w5-fdf4da-kserve-d4c5bbd6d-5xh26"] Oct 30 18:25:05.967370 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.967338 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:25:05.976022 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:05.975990 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/premium-w5-fdf4da-kserve-8577fffc6d-gkfzv"] Oct 30 18:25:07.308672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:07.308641 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" path="/var/lib/kubelet/pods/5bff7d2b-a767-434f-83c8-aa847746f21c/volumes" Oct 30 18:25:07.309168 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:07.309153 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" path="/var/lib/kubelet/pods/b42b2b39-1e15-41c7-82ae-d902239de481/volumes" Oct 30 18:25:07.309564 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:07.309550 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d367b5f9-6441-46e7-968c-11034679ef97" path="/var/lib/kubelet/pods/d367b5f9-6441-46e7-968c-11034679ef97/volumes" Oct 30 18:25:10.457253 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:10.457216 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:25:26.870569 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:26.870531 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:25:26.873732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:26.870900 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" containerID="cri-o://e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771" gracePeriod=30 Oct 30 18:25:27.767907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:27.767868 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:25:27.768146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:27.768120 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" containerID="cri-o://0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514" gracePeriod=30 Oct 30 18:25:32.177839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.177805 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:25:32.200220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.200183 5024 generic.go:356] "Generic (PLEG): container finished" podID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerID="e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771" exitCode=0 Oct 30 18:25:32.200372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.200228 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" Oct 30 18:25:32.200372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.200245 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" event={"ID":"52641dc2-7517-4aff-aa1f-760c3f98f00e","Type":"ContainerDied","Data":"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771"} Oct 30 18:25:32.200372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.200299 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q" event={"ID":"52641dc2-7517-4aff-aa1f-760c3f98f00e","Type":"ContainerDied","Data":"5ebe6db76c2071ff95dcd3648210723511407a897a82846584ef0cac5313823f"} Oct 30 18:25:32.200372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.200322 5024 scope.go:122] "RemoveContainer" containerID="e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771" Oct 30 18:25:32.212247 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.212216 5024 scope.go:122] "RemoveContainer" containerID="e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771" Oct 30 18:25:32.212530 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:32.212508 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771\": container with ID starting with e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771 not found: ID does not exist" containerID="e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771" Oct 30 18:25:32.212618 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.212541 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771"} err="failed to get container status \"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771\": rpc error: code = NotFound desc = could not find container \"e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771\": container with ID starting with e93896dbbf4f4cf97dcc86c8a9c4bb13279204a5caea59c07ba298b30ec4e771 not found: ID does not exist" Oct 30 18:25:32.293304 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293273 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293367 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293399 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293497 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293536 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293574 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293716 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293747 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.293867 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293809 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:32.293924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293884 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:32.293924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.293903 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\") pod \"52641dc2-7517-4aff-aa1f-760c3f98f00e\" (UID: \"52641dc2-7517-4aff-aa1f-760c3f98f00e\") " Oct 30 18:25:32.294089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294063 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:32.294157 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294080 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:32.294313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294294 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:32.294923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294900 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52641dc2-7517-4aff-aa1f-760c3f98f00e-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.295006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294929 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.295006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294944 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.295006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294961 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.295006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.294974 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.296274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.296249 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:32.296355 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.296274 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:32.296402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.296364 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:32.296402 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.296389 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78" pod "52641dc2-7517-4aff-aa1f-760c3f98f00e" (UID: "52641dc2-7517-4aff-aa1f-760c3f98f00e"). InnerVolumeSpecName "kube-api-access-p8l78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:32.396274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.396242 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.396274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.396271 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.396274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.396280 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8l78\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-kube-api-access-p8l78\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.396505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.396288 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52641dc2-7517-4aff-aa1f-760c3f98f00e-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:32.548456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.548423 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:25:32.554554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:32.554529 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-5nk9q"] Oct 30 18:25:33.080326 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.080295 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:25:33.108734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.108698 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.108928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.108772 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.108928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.108847 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.108928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.108913 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109035 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109082 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109120 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109181 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109235 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\") pod \"cd744c62-c781-4a12-8c9d-e789b39e50cd\" (UID: \"cd744c62-c781-4a12-8c9d-e789b39e50cd\") " Oct 30 18:25:33.109545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109485 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.109684 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109660 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.109998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.109971 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.110436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.110411 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.110531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.110442 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.110531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.110457 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.110717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.110693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:33.110937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.110910 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.114588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.114442 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "kube-api-access-nd7qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.114588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.114553 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.114773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.114688 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.115560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.115533 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo" pod "cd744c62-c781-4a12-8c9d-e789b39e50cd" (UID: "cd744c62-c781-4a12-8c9d-e789b39e50cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:33.210391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.210355 5024 generic.go:356] "Generic (PLEG): container finished" podID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerID="0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514" exitCode=0 Oct 30 18:25:33.210925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.210404 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" Oct 30 18:25:33.210925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.210442 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" event={"ID":"cd744c62-c781-4a12-8c9d-e789b39e50cd","Type":"ContainerDied","Data":"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514"} Oct 30 18:25:33.210925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.210506 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq" event={"ID":"cd744c62-c781-4a12-8c9d-e789b39e50cd","Type":"ContainerDied","Data":"f46af2e66666053d2d6a8107b96121495d54f0e8f5aa1fd279d4b170cdab4383"} Oct 30 18:25:33.210925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.210530 5024 scope.go:122] "RemoveContainer" containerID="0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514" Oct 30 18:25:33.211918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211899 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7qv\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-kube-api-access-nd7qv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.212036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211921 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.212036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211931 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd744c62-c781-4a12-8c9d-e789b39e50cd-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.212036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211939 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.212036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211948 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd744c62-c781-4a12-8c9d-e789b39e50cd-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.212036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.211960 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd744c62-c781-4a12-8c9d-e789b39e50cd-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.226703 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.226682 5024 scope.go:122] "RemoveContainer" containerID="0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514" Oct 30 18:25:33.227015 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:33.226984 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514\": container with ID starting with 0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514 not found: ID does not exist" containerID="0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514" Oct 30 18:25:33.227086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.227024 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514"} err="failed to get container status \"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514\": rpc error: code = NotFound desc = could not find container \"0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514\": container with ID starting with 0b1381ff0a46bd5697e15c2d165201255beaa341b61fb2f27db4a7c9b7352514 not found: ID does not exist" Oct 30 18:25:33.269204 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.269168 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:25:33.274808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.274778 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-42fbq"] Oct 30 18:25:33.308549 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.308471 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" path="/var/lib/kubelet/pods/52641dc2-7517-4aff-aa1f-760c3f98f00e/volumes" Oct 30 18:25:33.309153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:33.309136 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" path="/var/lib/kubelet/pods/cd744c62-c781-4a12-8c9d-e789b39e50cd/volumes" Oct 30 18:25:34.467964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.466397 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:25:34.467964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.466744 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" podUID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" containerName="maas-api" containerID="cri-o://4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4" gracePeriod=30 Oct 30 18:25:34.867966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.865816 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:25:34.995673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.990307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\") pod \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " Oct 30 18:25:34.995673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.990548 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\") pod \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " Oct 30 18:25:34.995673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.990586 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\") pod \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " Oct 30 18:25:34.995673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:34.990635 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\") pod \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\" (UID: \"c4bfc68b-7fca-4862-9557-ac06eb693fb0\") " Oct 30 18:25:35.037410 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.037011 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca" pod "c4bfc68b-7fca-4862-9557-ac06eb693fb0" (UID: "c4bfc68b-7fca-4862-9557-ac06eb693fb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:35.053617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.042515 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs" pod "c4bfc68b-7fca-4862-9557-ac06eb693fb0" (UID: "c4bfc68b-7fca-4862-9557-ac06eb693fb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:35.053617 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.047422 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v" pod "c4bfc68b-7fca-4862-9557-ac06eb693fb0" (UID: "c4bfc68b-7fca-4862-9557-ac06eb693fb0"). InnerVolumeSpecName "kube-api-access-h824v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:35.085629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.071728 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls" pod "c4bfc68b-7fca-4862-9557-ac06eb693fb0" (UID: "c4bfc68b-7fca-4862-9557-ac06eb693fb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:35.098270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.097271 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:35.098270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.097309 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4bfc68b-7fca-4862-9557-ac06eb693fb0-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:35.098270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.097325 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4bfc68b-7fca-4862-9557-ac06eb693fb0-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:35.098270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.097342 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h824v\" (UniqueName: \"kubernetes.io/projected/c4bfc68b-7fca-4862-9557-ac06eb693fb0-kube-api-access-h824v\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:35.229457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.229416 5024 generic.go:356] "Generic (PLEG): container finished" podID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" containerID="4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4" exitCode=0 Oct 30 18:25:35.229457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.229463 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" Oct 30 18:25:35.229758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.229503 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" event={"ID":"c4bfc68b-7fca-4862-9557-ac06eb693fb0","Type":"ContainerDied","Data":"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4"} Oct 30 18:25:35.229758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.229573 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5" event={"ID":"c4bfc68b-7fca-4862-9557-ac06eb693fb0","Type":"ContainerDied","Data":"4b42da46bdd16831308059ea1a4160eecc1bec0b4cbff40326f974c0829ae22e"} Oct 30 18:25:35.229758 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.229646 5024 scope.go:122] "RemoveContainer" containerID="4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4" Oct 30 18:25:35.245469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.245438 5024 scope.go:122] "RemoveContainer" containerID="4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4" Oct 30 18:25:35.245930 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:35.245858 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4\": container with ID starting with 4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4 not found: ID does not exist" containerID="4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4" Oct 30 18:25:35.245930 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.245900 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4"} err="failed to get container status \"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4\": rpc error: code = NotFound desc = could not find container \"4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4\": container with ID starting with 4a7c13fc61779c8120d1f3ed6a88b0de6d7ec7189eb63c0a3ef59e7170751de4 not found: ID does not exist" Oct 30 18:25:35.290711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.290678 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:25:35.321883 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:35.321841 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-fdf4da-6cf96f798c-bx9p5"] Oct 30 18:25:37.307691 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:37.307568 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" path="/var/lib/kubelet/pods/c4bfc68b-7fca-4862-9557-ac06eb693fb0/volumes" Oct 30 18:25:40.463891 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:40.463850 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:25:41.275336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:41.275300 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:25:41.289267 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:41.289221 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" containerID="cri-o://de151b65290dc1a7246c8810035fafd92382432bb3d3cdabe3d3da168ffa5514" gracePeriod=30 Oct 30 18:25:43.619172 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.619134 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:25:43.620266 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.620231 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" containerID="cri-o://038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" gracePeriod=30 Oct 30 18:25:43.630046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.630012 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:25:43.630293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.630267 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" containerID="cri-o://6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" gracePeriod=30 Oct 30 18:25:43.639369 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.639338 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:25:43.639589 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:43.639562 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" containerID="cri-o://ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" gracePeriod=30 Oct 30 18:25:46.484623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:46.478002 5024 generic.go:356] "Generic (PLEG): container finished" podID="75488f67-3857-4558-9a91-e85da8d88ea4" containerID="de151b65290dc1a7246c8810035fafd92382432bb3d3cdabe3d3da168ffa5514" exitCode=0 Oct 30 18:25:46.484623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:46.478214 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" event={"ID":"75488f67-3857-4558-9a91-e85da8d88ea4","Type":"ContainerDied","Data":"de151b65290dc1a7246c8810035fafd92382432bb3d3cdabe3d3da168ffa5514"} Oct 30 18:25:46.932198 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:46.931623 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:46.932198 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:46.931701 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:47.192622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.192502 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:47.192622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.192559 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:47.201118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.201093 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:25:47.218333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.218307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.218663 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.218639 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:47.218819 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.218802 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219000 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219174 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219423 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219403 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219076 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:47.219637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219250 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:47.219637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219630 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219753 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219731 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219780 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.219892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.219866 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\") pod \"75488f67-3857-4558-9a91-e85da8d88ea4\" (UID: \"75488f67-3857-4558-9a91-e85da8d88ea4\") " Oct 30 18:25:47.220368 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.220347 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:47.220868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.220846 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:47.221095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.221078 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.221179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.221100 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.221179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.221114 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.221179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.221127 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75488f67-3857-4558-9a91-e85da8d88ea4-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.221179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.221140 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.222238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.222210 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:47.222320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.222243 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "kube-api-access-ctsk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:47.222533 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.222510 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:47.222821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.222802 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy" pod "75488f67-3857-4558-9a91-e85da8d88ea4" (UID: "75488f67-3857-4558-9a91-e85da8d88ea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:47.324666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.322521 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.324666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.322547 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsk2\" (UniqueName: \"kubernetes.io/projected/75488f67-3857-4558-9a91-e85da8d88ea4-kube-api-access-ctsk2\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.324666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.322557 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75488f67-3857-4558-9a91-e85da8d88ea4-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.324666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.322567 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75488f67-3857-4558-9a91-e85da8d88ea4-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:47.575626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.571286 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" event={"ID":"75488f67-3857-4558-9a91-e85da8d88ea4","Type":"ContainerDied","Data":"b6c3c0743b4fddcbaf1733396cb8e3770ea4e81b67a9570671ffa49521663c5a"} Oct 30 18:25:47.575626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.571366 5024 scope.go:122] "RemoveContainer" containerID="de151b65290dc1a7246c8810035fafd92382432bb3d3cdabe3d3da168ffa5514" Oct 30 18:25:47.575626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.571524 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p" Oct 30 18:25:47.771843 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.771762 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:25:47.818624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:47.813492 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7kd8p"] Oct 30 18:25:49.233678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.233258 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:25:49.311232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.308812 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:25:49.315026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.314997 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" path="/var/lib/kubelet/pods/75488f67-3857-4558-9a91-e85da8d88ea4/volumes" Oct 30 18:25:49.390138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.390094 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.414400 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414369 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414417 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414460 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414484 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414534 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414574 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.414697 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414626 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.415072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414734 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.415072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414789 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.415072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.414893 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.415732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415700 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.415865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415780 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.415865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415834 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.416153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415862 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.416153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415920 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\") pod \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\" (UID: \"1ae9bd74-c688-4ba5-929d-cbe2ec877509\") " Oct 30 18:25:49.416153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.415992 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.416153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.416101 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\") pod \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\" (UID: \"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b\") " Oct 30 18:25:49.416372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.401887 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.418391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.417209 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.418391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.417435 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.418391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.417863 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.419103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.419002 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.419235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.419210 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.419311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.419222 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.419581 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.419419 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.420051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.419918 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.420144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.420095 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.423823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.422700 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.423823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.422962 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.423823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.423060 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.423823 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.423089 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.435290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.424145 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.435290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.424752 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "kube-api-access-hqggw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.435290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.431462 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "kube-api-access-rdf9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.435290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.434471 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token" pod "1ae9bd74-c688-4ba5-929d-cbe2ec877509" (UID: "1ae9bd74-c688-4ba5-929d-cbe2ec877509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.441752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.441706 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo" pod "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" (UID: "0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.444795 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.444774 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520315 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520348 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520364 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520377 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520390 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520409 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520403 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520417 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqggw\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-kube-api-access-hqggw\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520437 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520453 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520466 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520478 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520489 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520501 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520512 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae9bd74-c688-4ba5-929d-cbe2ec877509-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520525 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdf9v\" (UniqueName: \"kubernetes.io/projected/1ae9bd74-c688-4ba5-929d-cbe2ec877509-kube-api-access-rdf9v\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520537 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.520891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.520548 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.538631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.531393 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg"] Oct 30 18:25:49.538631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.531584 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" containerName="maas-api" containerID="cri-o://4504475e7f38a204e16708c6df69dd9529818cd9f0fcc6540050475e4ac4159e" gracePeriod=30 Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625130 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625221 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625253 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625314 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625385 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625437 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625468 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625518 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.626624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.625571 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\") pod \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\" (UID: \"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc\") " Oct 30 18:25:49.637870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.637243 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.637870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.637434 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.637870 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.637722 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.640000 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.639926 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.640482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.640438 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.717673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.663107 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "kube-api-access-xl7p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.717673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.688795 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.717673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.690757 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.717673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.690846 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy" pod "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" (UID: "6eb2a895-bf9e-4951-8900-e3aae2e6c1fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.736764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.736729 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.736998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.736982 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737083 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737176 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737260 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737341 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737420 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737497 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.737586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.737575 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7p9\" (UniqueName: \"kubernetes.io/projected/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc-kube-api-access-xl7p9\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.781690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:49.776463 5024 scope.go:122] "RemoveContainer" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.094717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.034211 5024 generic.go:356] "Generic (PLEG): container finished" podID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" exitCode=0 Oct 30 18:25:50.094717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.034530 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" event={"ID":"1ae9bd74-c688-4ba5-929d-cbe2ec877509","Type":"ContainerDied","Data":"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510"} Oct 30 18:25:50.094717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.034587 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" event={"ID":"1ae9bd74-c688-4ba5-929d-cbe2ec877509","Type":"ContainerDied","Data":"4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0"} Oct 30 18:25:50.094717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.034644 5024 scope.go:122] "RemoveContainer" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.094717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.034790 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d" Oct 30 18:25:50.150721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.130795 5024 generic.go:356] "Generic (PLEG): container finished" podID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" exitCode=0 Oct 30 18:25:50.150721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.130987 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" event={"ID":"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b","Type":"ContainerDied","Data":"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d"} Oct 30 18:25:50.150721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.131059 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" event={"ID":"0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b","Type":"ContainerDied","Data":"24792b7f5a837be0c628c46d716914b43d1bb24e906afdbf7ba2758c25644e1a"} Oct 30 18:25:50.150721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.131419 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" Oct 30 18:25:50.230637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.184671 5024 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:50.230637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.184740 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.272335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.270828 5024 generic.go:356] "Generic (PLEG): container finished" podID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerID="ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" exitCode=0 Oct 30 18:25:50.272335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.270900 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" event={"ID":"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc","Type":"ContainerDied","Data":"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e"} Oct 30 18:25:50.272335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.270948 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" event={"ID":"6eb2a895-bf9e-4951-8900-e3aae2e6c1fc","Type":"ContainerDied","Data":"23f6db9acfa129c962af89034f8538b4cd078890b5d21acf73070c5c41b70e13"} Oct 30 18:25:50.272335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.271152 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf" Oct 30 18:25:50.396371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.384576 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.118:8443/health\": dial tcp 10.132.0.118:8443: connect: connection refused" Oct 30 18:25:50.738626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.735919 5024 scope.go:122] "RemoveContainer" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.747489 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510\": container with ID starting with 6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510 not found: ID does not exist" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.747547 5024 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510\": container with ID starting with 6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510 not found: ID does not exist" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.747658 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510"} err="failed to get container status \"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510\": rpc error: code = NotFound desc = could not find container \"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510\": container with ID starting with 6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510 not found: ID does not exist" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.747685 5024 scope.go:122] "RemoveContainer" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.756660 5024 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d_openshift-ingress_1ae9bd74-c688-4ba5-929d-cbe2ec877509_0 in pod sandbox 4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0 from index: no such id: '6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510'" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.756721 5024 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d_openshift-ingress_1ae9bd74-c688-4ba5-929d-cbe2ec877509_0 in pod sandbox 4f558e73f5a4ff465907f4e598606091dafc5cf90b27985f37dff2e6b244dbe0 from index: no such id: '6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510'" containerID="6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510" Oct 30 18:25:50.764554 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.756752 5024 scope.go:122] "RemoveContainer" containerID="ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" Oct 30 18:25:50.770640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.770567 5024 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510\": container with ID starting with 6fc3693957c666db2296943f1a04e0ae6cc17af59287c5068ffbaafbbb6e8510 not found: ID does not exist" Oct 30 18:25:50.831635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.816724 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.914709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.914659 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:25:50.920641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.920590 5024 scope.go:122] "RemoveContainer" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" Oct 30 18:25:50.920761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.920703 5024 scope.go:122] "RemoveContainer" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" Oct 30 18:25:50.921196 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.921166 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d\": container with ID starting with 038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d not found: ID does not exist" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" Oct 30 18:25:50.921293 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.921203 5024 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d\": rpc error: code = NotFound desc = could not find container \"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d\": container with ID starting with 038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d not found: ID does not exist" containerID="038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d" Oct 30 18:25:50.925269 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.925226 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d"} err="failed to get container status \"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d\": rpc error: code = NotFound desc = could not find container \"038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d\": container with ID starting with 038851ec5084381b7cefb6c975f1490dde562ea4de4c47e73fab0df86a21547d not found: ID does not exist" Oct 30 18:25:50.925383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.925373 5024 scope.go:122] "RemoveContainer" containerID="ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" Oct 30 18:25:50.930909 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:25:50.930854 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e\": container with ID starting with ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e not found: ID does not exist" containerID="ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" Oct 30 18:25:50.931931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.931888 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e"} err="failed to get container status \"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e\": rpc error: code = NotFound desc = could not find container \"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e\": container with ID starting with ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e not found: ID does not exist" Oct 30 18:25:50.931931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.931932 5024 scope.go:122] "RemoveContainer" containerID="ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e" Oct 30 18:25:50.933217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.931093 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-5v26d"] Oct 30 18:25:50.933217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.932529 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e"} err="failed to get container status \"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e\": rpc error: code = NotFound desc = could not find container \"ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e\": container with ID starting with ef0d903dbe152e6913af04b33e99a5511c54a2013ad5ead8fc3e0c2dd30a869e not found: ID does not exist" Oct 30 18:25:50.940240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:50.940198 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:25:51.144631 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.127215 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-hvfvf"] Oct 30 18:25:51.637084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.637043 5024 generic.go:356] "Generic (PLEG): container finished" podID="756952ee-14ed-44aa-adb5-96cba8be023c" containerID="4504475e7f38a204e16708c6df69dd9529818cd9f0fcc6540050475e4ac4159e" exitCode=0 Oct 30 18:25:51.721660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.711150 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" path="/var/lib/kubelet/pods/0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b/volumes" Oct 30 18:25:51.721660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.711906 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" path="/var/lib/kubelet/pods/1ae9bd74-c688-4ba5-929d-cbe2ec877509/volumes" Oct 30 18:25:51.721660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.712360 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:25:51.721660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.712399 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z4tqf"] Oct 30 18:25:51.721660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.712425 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" event={"ID":"756952ee-14ed-44aa-adb5-96cba8be023c","Type":"ContainerDied","Data":"4504475e7f38a204e16708c6df69dd9529818cd9f0fcc6540050475e4ac4159e"} Oct 30 18:25:51.722275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.721934 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:25:51.740299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.740267 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\") pod \"756952ee-14ed-44aa-adb5-96cba8be023c\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " Oct 30 18:25:51.740579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.740552 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\") pod \"756952ee-14ed-44aa-adb5-96cba8be023c\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " Oct 30 18:25:51.740716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.740676 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca" pod "756952ee-14ed-44aa-adb5-96cba8be023c" (UID: "756952ee-14ed-44aa-adb5-96cba8be023c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:51.741299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.741272 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/756952ee-14ed-44aa-adb5-96cba8be023c-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.794623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.781088 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls" pod "756952ee-14ed-44aa-adb5-96cba8be023c" (UID: "756952ee-14ed-44aa-adb5-96cba8be023c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:51.842125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.842034 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\" (UniqueName: \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\") pod \"756952ee-14ed-44aa-adb5-96cba8be023c\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " Oct 30 18:25:51.842125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.842100 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\") pod \"756952ee-14ed-44aa-adb5-96cba8be023c\" (UID: \"756952ee-14ed-44aa-adb5-96cba8be023c\") " Oct 30 18:25:51.843005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.842728 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.844503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.844478 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs" pod "756952ee-14ed-44aa-adb5-96cba8be023c" (UID: "756952ee-14ed-44aa-adb5-96cba8be023c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:51.844614 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.844558 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684" pod "756952ee-14ed-44aa-adb5-96cba8be023c" (UID: "756952ee-14ed-44aa-adb5-96cba8be023c"). InnerVolumeSpecName "kube-api-access-qv684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:51.943981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.943949 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv684\" (UniqueName: \"kubernetes.io/projected/756952ee-14ed-44aa-adb5-96cba8be023c-kube-api-access-qv684\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.943981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:51.943978 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/756952ee-14ed-44aa-adb5-96cba8be023c-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:25:52.731991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:52.731908 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" Oct 30 18:25:52.731991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:52.731927 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg" event={"ID":"756952ee-14ed-44aa-adb5-96cba8be023c","Type":"ContainerDied","Data":"25b5bdc95d8386674d87cc92cea84b42c3f7adf0268bd582392317b2462d43bd"} Oct 30 18:25:52.732493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:52.732013 5024 scope.go:122] "RemoveContainer" containerID="4504475e7f38a204e16708c6df69dd9529818cd9f0fcc6540050475e4ac4159e" Oct 30 18:25:52.783350 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:52.783315 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg"] Oct 30 18:25:52.788978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:52.788948 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9af5623f-d7f774cdd-76zdg"] Oct 30 18:25:53.310386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:53.310351 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" path="/var/lib/kubelet/pods/6eb2a895-bf9e-4951-8900-e3aae2e6c1fc/volumes" Oct 30 18:25:53.310913 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:53.310896 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" path="/var/lib/kubelet/pods/756952ee-14ed-44aa-adb5-96cba8be023c/volumes" Oct 30 18:25:54.173817 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:54.173777 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:54.174268 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:54.173835 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.156771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.156730 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:55.156982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.156790 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.957224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.957159 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:25:55.958200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.957470 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" containerID="cri-o://9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962" gracePeriod=30 Oct 30 18:25:55.974243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.974188 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:25:55.974881 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:55.974837 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" containerID="cri-o://44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7" gracePeriod=30 Oct 30 18:25:56.019212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.019177 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:25:56.019642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.019610 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" containerID="cri-o://a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63" gracePeriod=30 Oct 30 18:25:56.766302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.766262 5024 generic.go:356] "Generic (PLEG): container finished" podID="dabfa326-cef6-471b-b332-686a2827c2c8" containerID="fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769" exitCode=0 Oct 30 18:25:56.766525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.766338 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerDied","Data":"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769"} Oct 30 18:25:56.766926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.766908 5024 scope.go:122] "RemoveContainer" containerID="fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769" Oct 30 18:25:56.768863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.768832 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:56.768955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.768906 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:56.768955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:56.768943 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:57.778651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.778615 5024 generic.go:356] "Generic (PLEG): container finished" podID="c16e7315-0881-41cc-898d-378727f3335d" containerID="6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92" exitCode=0 Oct 30 18:25:57.779164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.778695 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerDied","Data":"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92"} Oct 30 18:25:57.779239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.779203 5024 scope.go:122] "RemoveContainer" containerID="6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92" Oct 30 18:25:57.780994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.780974 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerStarted","Data":"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484"} Oct 30 18:25:57.781109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.781084 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:57.781165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.781152 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:57.781214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:57.781187 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:25:58.125718 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.125621 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:25:58.125897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.125875 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:25:58.125941 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.125922 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:25:58.543105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.542980 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:25:58.751626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.750715 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:25:58.924468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.924390 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerStarted","Data":"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5"} Oct 30 18:25:58.924988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.924531 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" containerID="cri-o://6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5" gracePeriod=30 Oct 30 18:25:58.925344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:58.925236 5024 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" secret="" err="secret \"e2e-mt-9af5623f-openshift-default-dockercfg-vxdf5\" not found" Oct 30 18:25:59.198975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:59.198877 5024 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:25:59.210290 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:59.210246 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 18:25:59.933531 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:25:59.933492 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" containerID="cri-o://30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484" gracePeriod=30 Oct 30 18:26:01.364437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.364069 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:26:01.490492 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.490462 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.491023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.491003 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.491219 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.491201 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.491726 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.491711 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.492206 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.492188 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.492947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.492928 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.493504 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.493485 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.494040 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.494024 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.490934 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.491556 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.492084 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.494562 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.492865 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.494624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.494405 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.495011 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.494994 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\") pod \"f925cf97-755f-4bd8-adfe-3d406b99270a\" (UID: \"f925cf97-755f-4bd8-adfe-3d406b99270a\") " Oct 30 18:26:01.495892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.495872 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.496899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.496312 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497669 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497693 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497709 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f925cf97-755f-4bd8-adfe-3d406b99270a-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497727 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497740 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f925cf97-755f-4bd8-adfe-3d406b99270a-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.497762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.497339 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:01.498447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.498418 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv" pod "f925cf97-755f-4bd8-adfe-3d406b99270a" (UID: "f925cf97-755f-4bd8-adfe-3d406b99270a"). InnerVolumeSpecName "kube-api-access-2rnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.502078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.502053 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:26:01.511101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.511071 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:26:01.600184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.600092 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.601793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.601763 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.602159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.602137 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.602638 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.602617 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.603140 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.603120 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.604486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.604457 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.604763 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.604744 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.605293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.601443 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.605293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.602306 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.605293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.602466 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.605293 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.603036 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.605652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.605630 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.606201 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606083 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:01.606362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606345 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.606759 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606724 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.606873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606773 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.606873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606824 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.606989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606896 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.606989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606934 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.606989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606962 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.607156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.606995 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.607156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.607038 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\") pod \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\" (UID: \"72ee49bb-4874-4b7c-832f-cccfddea8e9e\") " Oct 30 18:26:01.607156 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.607108 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\") pod \"3505cae6-b485-4d00-aaba-fe4655b66465\" (UID: \"3505cae6-b485-4d00-aaba-fe4655b66465\") " Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.607617 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608032 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608162 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608180 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608200 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608214 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608230 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnrv\" (UniqueName: \"kubernetes.io/projected/f925cf97-755f-4bd8-adfe-3d406b99270a-kube-api-access-2rnrv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608245 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608259 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f925cf97-755f-4bd8-adfe-3d406b99270a-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608272 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608352 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.608934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608359 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.608934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.608404 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.609203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.609003 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.609435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.609399 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.613118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.613065 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "kube-api-access-4bchf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.613852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.613813 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:01.614661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.614564 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.614661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.614652 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:01.614869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.614845 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "kube-api-access-4mhxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.614918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.614849 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo" pod "3505cae6-b485-4d00-aaba-fe4655b66465" (UID: "3505cae6-b485-4d00-aaba-fe4655b66465"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:01.617250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.617214 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token" pod "72ee49bb-4874-4b7c-832f-cccfddea8e9e" (UID: "72ee49bb-4874-4b7c-832f-cccfddea8e9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:01.710884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.710850 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhxk\" (UniqueName: \"kubernetes.io/projected/3505cae6-b485-4d00-aaba-fe4655b66465-kube-api-access-4mhxk\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711122 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711226 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3505cae6-b485-4d00-aaba-fe4655b66465-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711336 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711326 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711428 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711416 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711501 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711725 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bchf\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-kube-api-access-4bchf\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.711996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.711981 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.712084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.712074 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3505cae6-b485-4d00-aaba-fe4655b66465-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.712157 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.712148 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ee49bb-4874-4b7c-832f-cccfddea8e9e-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.712235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.712225 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3505cae6-b485-4d00-aaba-fe4655b66465-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.712563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.712549 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ee49bb-4874-4b7c-832f-cccfddea8e9e-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:01.952076 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.952035 5024 generic.go:356] "Generic (PLEG): container finished" podID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerID="44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7" exitCode=0 Oct 30 18:26:01.952380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.952354 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" event={"ID":"f925cf97-755f-4bd8-adfe-3d406b99270a","Type":"ContainerDied","Data":"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7"} Oct 30 18:26:01.952566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.952551 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" event={"ID":"f925cf97-755f-4bd8-adfe-3d406b99270a","Type":"ContainerDied","Data":"be473864a29d6090dd4e56440cee0346ef633fa46cf01efedcbfd08a895fb46a"} Oct 30 18:26:01.952719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.952703 5024 scope.go:122] "RemoveContainer" containerID="44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7" Oct 30 18:26:01.952990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.952972 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5" Oct 30 18:26:01.987095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.987060 5024 generic.go:356] "Generic (PLEG): container finished" podID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerID="a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63" exitCode=0 Oct 30 18:26:01.987513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.987488 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" event={"ID":"72ee49bb-4874-4b7c-832f-cccfddea8e9e","Type":"ContainerDied","Data":"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63"} Oct 30 18:26:01.988433 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.988409 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" event={"ID":"72ee49bb-4874-4b7c-832f-cccfddea8e9e","Type":"ContainerDied","Data":"a34b4e20252e1ecfe6efd00244afc8df1382adacd80740d99b0c34a562e48a61"} Oct 30 18:26:01.989155 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.989135 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q" Oct 30 18:26:01.990665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.990637 5024 generic.go:356] "Generic (PLEG): container finished" podID="3505cae6-b485-4d00-aaba-fe4655b66465" containerID="9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962" exitCode=0 Oct 30 18:26:01.990777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.990762 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" event={"ID":"3505cae6-b485-4d00-aaba-fe4655b66465","Type":"ContainerDied","Data":"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962"} Oct 30 18:26:01.990862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.990814 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" event={"ID":"3505cae6-b485-4d00-aaba-fe4655b66465","Type":"ContainerDied","Data":"cdc86b84b4ecd5c96716807ad908dbc17f9c74f04cf55e8c0530def6610537e4"} Oct 30 18:26:01.990964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:01.990931 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv" Oct 30 18:26:02.014195 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.014166 5024 scope.go:122] "RemoveContainer" containerID="44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7" Oct 30 18:26:02.015178 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:02.015153 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7\": container with ID starting with 44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7 not found: ID does not exist" containerID="44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7" Oct 30 18:26:02.015271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.015186 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7"} err="failed to get container status \"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7\": rpc error: code = NotFound desc = could not find container \"44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7\": container with ID starting with 44fd6cd42b7d55646814e11c6f2ead669b8da640a2152b99e7e5d3326e8db3d7 not found: ID does not exist" Oct 30 18:26:02.015271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.015206 5024 scope.go:122] "RemoveContainer" containerID="a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63" Oct 30 18:26:02.038562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.038530 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:26:02.038727 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.038549 5024 scope.go:122] "RemoveContainer" containerID="a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63" Oct 30 18:26:02.039077 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:02.039045 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63\": container with ID starting with a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63 not found: ID does not exist" containerID="a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63" Oct 30 18:26:02.039127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.039090 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63"} err="failed to get container status \"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63\": rpc error: code = NotFound desc = could not find container \"a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63\": container with ID starting with a8f0c64ae232263c7f54de50f0141c1a56b3e1da5d9be6aa1f5b0f9afbf95e63 not found: ID does not exist" Oct 30 18:26:02.039127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.039114 5024 scope.go:122] "RemoveContainer" containerID="9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962" Oct 30 18:26:02.071639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.069423 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-c4mn5"] Oct 30 18:26:02.077406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.077380 5024 scope.go:122] "RemoveContainer" containerID="9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962" Oct 30 18:26:02.077763 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:02.077738 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962\": container with ID starting with 9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962 not found: ID does not exist" containerID="9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962" Oct 30 18:26:02.077855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.077772 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962"} err="failed to get container status \"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962\": rpc error: code = NotFound desc = could not find container \"9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962\": container with ID starting with 9f790208258a9274bad3bfc3a8ea08755e1824d20517036cda9a23ec36e47962 not found: ID does not exist" Oct 30 18:26:02.100520 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.100453 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:26:02.106501 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.106464 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-rwrpv"] Oct 30 18:26:02.141634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.138762 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:26:02.165626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:02.156252 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-zsx9q"] Oct 30 18:26:03.312255 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:03.312199 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" path="/var/lib/kubelet/pods/3505cae6-b485-4d00-aaba-fe4655b66465/volumes" Oct 30 18:26:03.313072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:03.313049 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" path="/var/lib/kubelet/pods/72ee49bb-4874-4b7c-832f-cccfddea8e9e/volumes" Oct 30 18:26:03.313857 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:03.313834 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" path="/var/lib/kubelet/pods/f925cf97-755f-4bd8-adfe-3d406b99270a/volumes" Oct 30 18:26:04.270671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.270645 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:26:04.356732 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.356700 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.356789 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.356843 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.356872 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.356903 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357127 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357284 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357379 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\") pod \"c16e7315-0881-41cc-898d-378727f3335d\" (UID: \"c16e7315-0881-41cc-898d-378727f3335d\") " Oct 30 18:26:04.357624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357277 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.357624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357471 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.358729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.357926 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.358729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358244 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.358729 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358473 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:04.358988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358781 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.358988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358811 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.358988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358825 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.358988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358843 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16e7315-0881-41cc-898d-378727f3335d-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.358988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.358860 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.360177 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.360140 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.360177 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.360175 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.360871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.360851 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "kube-api-access-7674g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.361527 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.361507 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo" pod "c16e7315-0881-41cc-898d-378727f3335d" (UID: "c16e7315-0881-41cc-898d-378727f3335d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:04.460331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.460295 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16e7315-0881-41cc-898d-378727f3335d-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.460331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.460333 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16e7315-0881-41cc-898d-378727f3335d-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.460541 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.460350 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.460541 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:04.460366 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7674g\" (UniqueName: \"kubernetes.io/projected/c16e7315-0881-41cc-898d-378727f3335d-kube-api-access-7674g\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:05.022005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.021971 5024 generic.go:356] "Generic (PLEG): container finished" podID="c16e7315-0881-41cc-898d-378727f3335d" containerID="6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5" exitCode=0 Oct 30 18:26:05.022243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.022021 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" Oct 30 18:26:05.022243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.022056 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerDied","Data":"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5"} Oct 30 18:26:05.022243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.022122 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq" event={"ID":"c16e7315-0881-41cc-898d-378727f3335d","Type":"ContainerDied","Data":"fc11bc397e50f19b6c399a0c58a2b9cc09e30b1fc462e5e3daa10cc2a7b0fb07"} Oct 30 18:26:05.022243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.022146 5024 scope.go:122] "RemoveContainer" containerID="6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5" Oct 30 18:26:05.035669 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.035646 5024 scope.go:122] "RemoveContainer" containerID="6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92" Oct 30 18:26:05.046647 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.046583 5024 scope.go:122] "RemoveContainer" containerID="6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5" Oct 30 18:26:05.046967 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:05.046940 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5\": container with ID starting with 6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5 not found: ID does not exist" containerID="6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5" Oct 30 18:26:05.047125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.047095 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5"} err="failed to get container status \"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5\": rpc error: code = NotFound desc = could not find container \"6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5\": container with ID starting with 6f7612d2a828d854c967341b1e4d8774aea2b922f8e522b598c57cb19bcc43c5 not found: ID does not exist" Oct 30 18:26:05.047243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.047128 5024 scope.go:122] "RemoveContainer" containerID="6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92" Oct 30 18:26:05.047502 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:05.047481 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92\": container with ID starting with 6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92 not found: ID does not exist" containerID="6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92" Oct 30 18:26:05.047556 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.047510 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92"} err="failed to get container status \"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92\": rpc error: code = NotFound desc = could not find container \"6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92\": container with ID starting with 6bc80be7060f3f03c92df5cf34ee270ccef205a44ebe95798fd97627d4005b92 not found: ID does not exist" Oct 30 18:26:05.086837 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.086807 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:26:05.097869 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.097827 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-lwlnq"] Oct 30 18:26:05.307573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:05.307498 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16e7315-0881-41cc-898d-378727f3335d" path="/var/lib/kubelet/pods/c16e7315-0881-41cc-898d-378727f3335d/volumes" Oct 30 18:26:08.131092 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.131053 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:26:08.736978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.736947 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:26:08.933058 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933022 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933058 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933063 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933081 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933099 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933164 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933203 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933244 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933376 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933566 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933382 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:08.933746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933565 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:08.933746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933590 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:08.933746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933643 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\") pod \"dabfa326-cef6-471b-b332-686a2827c2c8\" (UID: \"dabfa326-cef6-471b-b332-686a2827c2c8\") " Oct 30 18:26:08.933938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.933904 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:08.934212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934171 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:08.934408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934388 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:08.934488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934416 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:08.934488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934431 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:08.934488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934444 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabfa326-cef6-471b-b332-686a2827c2c8-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:08.934488 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.934457 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:08.935815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.935786 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:08.935917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.935847 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:08.935917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.935859 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "kube-api-access-lzxgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:08.936028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:08.936018 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo" pod "dabfa326-cef6-471b-b332-686a2827c2c8" (UID: "dabfa326-cef6-471b-b332-686a2827c2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:09.035931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.035893 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabfa326-cef6-471b-b332-686a2827c2c8-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:09.035931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.035925 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabfa326-cef6-471b-b332-686a2827c2c8-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:09.035931 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.035937 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxgc\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-kube-api-access-lzxgc\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:09.036185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.035947 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabfa326-cef6-471b-b332-686a2827c2c8-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:09.092840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.092806 5024 generic.go:356] "Generic (PLEG): container finished" podID="dabfa326-cef6-471b-b332-686a2827c2c8" containerID="30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484" exitCode=0 Oct 30 18:26:09.093028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.092853 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" Oct 30 18:26:09.093028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.092887 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerDied","Data":"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484"} Oct 30 18:26:09.093028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.092945 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf" event={"ID":"dabfa326-cef6-471b-b332-686a2827c2c8","Type":"ContainerDied","Data":"3df79bfe269580253f1b6bcc1603d043bf3e4574e78d14beb9bac21a28dad3d3"} Oct 30 18:26:09.093028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.092968 5024 scope.go:122] "RemoveContainer" containerID="30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484" Oct 30 18:26:09.104670 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.104652 5024 scope.go:122] "RemoveContainer" containerID="fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769" Oct 30 18:26:09.115629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.115577 5024 scope.go:122] "RemoveContainer" containerID="30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484" Oct 30 18:26:09.116020 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:09.115995 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484\": container with ID starting with 30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484 not found: ID does not exist" containerID="30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484" Oct 30 18:26:09.116119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.116032 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484"} err="failed to get container status \"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484\": rpc error: code = NotFound desc = could not find container \"30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484\": container with ID starting with 30110978a85d76746d2c8c04ba330c2a8472f07553a53f249512ac190f30b484 not found: ID does not exist" Oct 30 18:26:09.116119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.116058 5024 scope.go:122] "RemoveContainer" containerID="fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769" Oct 30 18:26:09.116409 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:09.116388 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769\": container with ID starting with fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769 not found: ID does not exist" containerID="fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769" Oct 30 18:26:09.116455 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.116415 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769"} err="failed to get container status \"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769\": rpc error: code = NotFound desc = could not find container \"fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769\": container with ID starting with fdff00823a93f25d717016206f34679055e17de1fac97b150e920cd18fa86769 not found: ID does not exist" Oct 30 18:26:09.143962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.143929 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:26:09.154413 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.154383 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-5jfhf"] Oct 30 18:26:09.348642 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:09.348543 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" path="/var/lib/kubelet/pods/dabfa326-cef6-471b-b332-686a2827c2c8/volumes" Oct 30 18:26:10.456694 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:10.456648 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:26:10.788739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.788657 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:26:10.789313 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789300 5024 cpu_manager.go:414] "containerMap: removing container" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" Oct 30 18:26:10.789357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789315 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" Oct 30 18:26:10.789357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789336 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" Oct 30 18:26:10.789357 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789341 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789359 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789363 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789371 5024 cpu_manager.go:414] "containerMap: removing container" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789376 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789383 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" containerName="maas-api" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789387 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" containerName="maas-api" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789395 5024 cpu_manager.go:414] "containerMap: removing container" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789399 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789412 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789417 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789430 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789435 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789442 5024 cpu_manager.go:414] "containerMap: removing container" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789448 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789454 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:26:10.789456 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789458 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789467 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789471 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789479 5024 cpu_manager.go:414] "containerMap: removing container" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" containerName="maas-api" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789483 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" containerName="maas-api" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789498 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789503 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789511 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789515 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789526 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789530 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789546 5024 cpu_manager.go:414] "containerMap: removing container" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789550 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789706 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ee49bb-4874-4b7c-832f-cccfddea8e9e" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789718 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f575041-4a4e-42d0-a45f-c6f2f8a5fb3b" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789728 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f925cf97-755f-4bd8-adfe-3d406b99270a" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789735 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd744c62-c781-4a12-8c9d-e789b39e50cd" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789740 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bfc68b-7fca-4862-9557-ac06eb693fb0" containerName="maas-api" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789746 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="52641dc2-7517-4aff-aa1f-760c3f98f00e" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789752 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d367b5f9-6441-46e7-968c-11034679ef97" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789757 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3505cae6-b485-4d00-aaba-fe4655b66465" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789762 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789767 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789773 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42b2b39-1e15-41c7-82ae-d902239de481" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789778 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="756952ee-14ed-44aa-adb5-96cba8be023c" containerName="maas-api" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789783 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789790 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bff7d2b-a767-434f-83c8-aa847746f21c" containerName="main" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789797 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb2a895-bf9e-4951-8900-e3aae2e6c1fc" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789817 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae9bd74-c688-4ba5-929d-cbe2ec877509" containerName="istio-proxy" Oct 30 18:26:10.789925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.789823 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="75488f67-3857-4558-9a91-e85da8d88ea4" containerName="istio-proxy" Oct 30 18:26:10.794983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.794965 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.812856 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.812828 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-f0837a-serving-cert\"" Oct 30 18:26:10.813275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.813238 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:26:10.876784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.876729 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.877004 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.876981 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.877119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.877098 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccb7\" (UniqueName: \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.877252 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.877229 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.978662 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.978609 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.978826 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.978685 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccb7\" (UniqueName: \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.978826 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.978765 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.978935 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.978853 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.979468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.979440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.981245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.981226 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:10.981330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:10.981272 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:11.003474 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:11.003441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccb7\" (UniqueName: \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\") pod \"maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:11.119807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:11.119725 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:11.473056 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:26:11.473018 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bc7c22_41a1_46cb_b9ca_d8e5ec8c6d7a.slice/crio-2a10b3b200a22b8777e87152a94dbba6ffe31b85255f345521be5d360958810c WatchSource:0}: Error finding container 2a10b3b200a22b8777e87152a94dbba6ffe31b85255f345521be5d360958810c: Status 404 returned error can't find the container with id 2a10b3b200a22b8777e87152a94dbba6ffe31b85255f345521be5d360958810c Oct 30 18:26:11.473509 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:11.473188 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:26:12.150934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:12.150893 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" event={"ID":"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a","Type":"ContainerStarted","Data":"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2"} Oct 30 18:26:12.151142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:12.150966 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" event={"ID":"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a","Type":"ContainerStarted","Data":"2a10b3b200a22b8777e87152a94dbba6ffe31b85255f345521be5d360958810c"} Oct 30 18:26:12.152153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:12.152128 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:12.193709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:12.193661 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" podStartSLOduration=1.871029585 podStartE2EDuration="2.193646327s" totalImagesPullingTime="322.616742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:12.189219242 +0000 UTC m=+1523.009931019" watchObservedRunningTime="2026-10-30 18:26:12.193646327 +0000 UTC m=+1523.014358102" Oct 30 18:26:19.170892 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:19.170862 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:26:39.325613 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:39.325565 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:26:40.490136 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:40.490089 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:26:51.142096 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:51.142038 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:26:51.143902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:51.143864 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerName="maas-api" containerID="cri-o://296b17e01c60d2c472b55b4a52a1eb8f1fdeedf0184cca6070bc3aab22c19fb2" gracePeriod=30 Oct 30 18:26:51.830947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:51.830650 5024 generic.go:356] "Generic (PLEG): container finished" podID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerID="296b17e01c60d2c472b55b4a52a1eb8f1fdeedf0184cca6070bc3aab22c19fb2" exitCode=0 Oct 30 18:26:51.830947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:51.830876 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" event={"ID":"bf4ddec7-9b80-404c-b783-a8d6f36f8074","Type":"ContainerDied","Data":"296b17e01c60d2c472b55b4a52a1eb8f1fdeedf0184cca6070bc3aab22c19fb2"} Oct 30 18:26:51.900387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:51.900347 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.119:8443/health\": dial tcp 10.132.0.119:8443: connect: connection refused" Oct 30 18:26:52.614625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.610939 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:52.639616 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.636792 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:26:52.707623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.707134 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\") pod \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " Oct 30 18:26:52.707623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.707247 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\") pod \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " Oct 30 18:26:52.707623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.707274 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\") pod \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " Oct 30 18:26:52.707623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.707502 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\" (UniqueName: \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\") pod \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\" (UID: \"bf4ddec7-9b80-404c-b783-a8d6f36f8074\") " Oct 30 18:26:52.709630 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.709572 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca" pod "bf4ddec7-9b80-404c-b783-a8d6f36f8074" (UID: "bf4ddec7-9b80-404c-b783-a8d6f36f8074"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:52.899686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.822551 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ddec7-9b80-404c-b783-a8d6f36f8074-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:52.899686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.860528 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs" pod "bf4ddec7-9b80-404c-b783-a8d6f36f8074" (UID: "bf4ddec7-9b80-404c-b783-a8d6f36f8074"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:52.899686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.860692 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f" pod "bf4ddec7-9b80-404c-b783-a8d6f36f8074" (UID: "bf4ddec7-9b80-404c-b783-a8d6f36f8074"). InnerVolumeSpecName "kube-api-access-dlh5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:53.004619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:52.994337 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls" pod "bf4ddec7-9b80-404c-b783-a8d6f36f8074" (UID: "bf4ddec7-9b80-404c-b783-a8d6f36f8074"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:53.037622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.036804 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh5f\" (UniqueName: \"kubernetes.io/projected/bf4ddec7-9b80-404c-b783-a8d6f36f8074-kube-api-access-dlh5f\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:53.037622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.036853 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:53.037622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.036870 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf4ddec7-9b80-404c-b783-a8d6f36f8074-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:53.062624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.054525 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" event={"ID":"bf4ddec7-9b80-404c-b783-a8d6f36f8074","Type":"ContainerDied","Data":"4997a0362479203a8c4f0dc317840cb19254b2b9b921fe7b9c86fa2e63d32fc8"} Oct 30 18:26:53.062624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.055853 5024 scope.go:122] "RemoveContainer" containerID="296b17e01c60d2c472b55b4a52a1eb8f1fdeedf0184cca6070bc3aab22c19fb2" Oct 30 18:26:53.062624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.054875 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp" Oct 30 18:26:53.156678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.145268 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:26:53.178075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.177452 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-295497-65d9c846cc-khbfp"] Oct 30 18:26:53.320677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:53.319451 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" path="/var/lib/kubelet/pods/bf4ddec7-9b80-404c-b783-a8d6f36f8074/volumes" Oct 30 18:26:54.351461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.351415 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" podUID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" containerName="main" containerID="cri-o://4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623" gracePeriod=45 Oct 30 18:26:54.697506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.697475 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:26:54.776519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.776480 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.776773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.776639 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.776773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.776685 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.776939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.776921 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.777002 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.776949 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:54.777056 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777025 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\" (UniqueName: \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.777114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777079 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\") pod \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\" (UID: \"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411\") " Oct 30 18:26:54.777479 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777448 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:54.777751 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777688 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:54.777871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777813 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.777871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.777837 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.779519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.779497 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:54.781057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.781029 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:54.781057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.781039 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29" pod "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" (UID: "ff46dbfd-1e57-47fc-ae2b-c5d3920cf411"). InnerVolumeSpecName "kube-api-access-d9t29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:54.878926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.878891 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9t29\" (UniqueName: \"kubernetes.io/projected/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-kube-api-access-d9t29\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.878926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.878920 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.878926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.878930 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:54.879163 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:54.878939 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:26:55.070817 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.070723 5024 generic.go:356] "Generic (PLEG): container finished" podID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" containerID="4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623" exitCode=0 Oct 30 18:26:55.070817 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.070775 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" Oct 30 18:26:55.070817 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.070784 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" event={"ID":"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411","Type":"ContainerDied","Data":"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623"} Oct 30 18:26:55.071112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.070834 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm" event={"ID":"ff46dbfd-1e57-47fc-ae2b-c5d3920cf411","Type":"ContainerDied","Data":"f9bacf4fa6cb996bde9dcb24561fc1d33aabd315048dc0b23d25d1173ee0fe33"} Oct 30 18:26:55.071112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.070857 5024 scope.go:122] "RemoveContainer" containerID="4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623" Oct 30 18:26:55.082621 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.082586 5024 scope.go:122] "RemoveContainer" containerID="4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623" Oct 30 18:26:55.083018 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:26:55.082998 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623\": container with ID starting with 4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623 not found: ID does not exist" containerID="4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623" Oct 30 18:26:55.083070 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.083028 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623"} err="failed to get container status \"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623\": rpc error: code = NotFound desc = could not find container \"4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623\": container with ID starting with 4e1f6a9459f9efe2867a010b10a16f71ffb654c4e7a86f2022b7e848dc1e7623 not found: ID does not exist" Oct 30 18:26:55.123041 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.123009 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:26:55.133985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.133958 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fb0abbf7/e2e-lifecycle-model-fb0abbf7-kserve-78974b89bf-zcknm"] Oct 30 18:26:55.307008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:26:55.306972 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" path="/var/lib/kubelet/pods/ff46dbfd-1e57-47fc-ae2b-c5d3920cf411/volumes" Oct 30 18:27:10.457820 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:27:10.457737 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:27:17.247802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.247761 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:27:17.248283 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.248012 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" podUID="c1db325a-4c0b-4381-8a92-16eda65f4579" containerName="maas-api" containerID="cri-o://9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6" gracePeriod=30 Oct 30 18:27:17.797093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.795964 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:27:17.895734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.895689 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\" (UniqueName: \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\") pod \"c1db325a-4c0b-4381-8a92-16eda65f4579\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " Oct 30 18:27:17.895734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.895737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\") pod \"c1db325a-4c0b-4381-8a92-16eda65f4579\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " Oct 30 18:27:17.896000 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.895854 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\") pod \"c1db325a-4c0b-4381-8a92-16eda65f4579\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " Oct 30 18:27:17.896000 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.895914 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\") pod \"c1db325a-4c0b-4381-8a92-16eda65f4579\" (UID: \"c1db325a-4c0b-4381-8a92-16eda65f4579\") " Oct 30 18:27:17.896144 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.896115 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca" pod "c1db325a-4c0b-4381-8a92-16eda65f4579" (UID: "c1db325a-4c0b-4381-8a92-16eda65f4579"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:17.896740 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.896703 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1db325a-4c0b-4381-8a92-16eda65f4579-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.898637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.898611 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs" pod "c1db325a-4c0b-4381-8a92-16eda65f4579" (UID: "c1db325a-4c0b-4381-8a92-16eda65f4579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:17.898789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.898741 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb" pod "c1db325a-4c0b-4381-8a92-16eda65f4579" (UID: "c1db325a-4c0b-4381-8a92-16eda65f4579"). InnerVolumeSpecName "kube-api-access-64mnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:17.899031 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.899007 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls" pod "c1db325a-4c0b-4381-8a92-16eda65f4579" (UID: "c1db325a-4c0b-4381-8a92-16eda65f4579"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:17.998637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.998517 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mnb\" (UniqueName: \"kubernetes.io/projected/c1db325a-4c0b-4381-8a92-16eda65f4579-kube-api-access-64mnb\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.998637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.998545 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:17.998637 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:17.998555 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1db325a-4c0b-4381-8a92-16eda65f4579-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:18.339189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.339097 5024 generic.go:356] "Generic (PLEG): container finished" podID="c1db325a-4c0b-4381-8a92-16eda65f4579" containerID="9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6" exitCode=0 Oct 30 18:27:18.339189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.339148 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" Oct 30 18:27:18.339189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.339156 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" event={"ID":"c1db325a-4c0b-4381-8a92-16eda65f4579","Type":"ContainerDied","Data":"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6"} Oct 30 18:27:18.339747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.339208 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9" event={"ID":"c1db325a-4c0b-4381-8a92-16eda65f4579","Type":"ContainerDied","Data":"e4fefc22e003221830a2ffdd46538606dcbf2c51a9dd610bd1ca018befc82fdf"} Oct 30 18:27:18.339747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.339238 5024 scope.go:122] "RemoveContainer" containerID="9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6" Oct 30 18:27:18.351579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.351559 5024 scope.go:122] "RemoveContainer" containerID="9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6" Oct 30 18:27:18.351966 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:27:18.351940 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6\": container with ID starting with 9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6 not found: ID does not exist" containerID="9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6" Oct 30 18:27:18.352065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.351973 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6"} err="failed to get container status \"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6\": rpc error: code = NotFound desc = could not find container \"9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6\": container with ID starting with 9ae7ec04466e3bfda6cf94990862e5f1311f523ceee85ac2efd609bb52fdb8b6 not found: ID does not exist" Oct 30 18:27:18.389641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.389592 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:27:18.399215 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:18.399184 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9066aa-5f7fdc9858-s5ff9"] Oct 30 18:27:19.307183 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:19.307150 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1db325a-4c0b-4381-8a92-16eda65f4579" path="/var/lib/kubelet/pods/c1db325a-4c0b-4381-8a92-16eda65f4579/volumes" Oct 30 18:27:24.739936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:24.739903 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:27:24.740525 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:24.740167 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" podUID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" containerName="maas-api" containerID="cri-o://af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b" gracePeriod=30 Oct 30 18:27:25.327042 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.327015 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:27:25.416800 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416760 5024 generic.go:356] "Generic (PLEG): container finished" podID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" containerID="af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b" exitCode=0 Oct 30 18:27:25.417005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416810 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" Oct 30 18:27:25.417005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416859 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" event={"ID":"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d","Type":"ContainerDied","Data":"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b"} Oct 30 18:27:25.417005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416892 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\") pod \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " Oct 30 18:27:25.417005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416916 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp" event={"ID":"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d","Type":"ContainerDied","Data":"30a91f22719a0770071fa660da2e3ffee7b219edc8c92caa3ebc9fbe9a89a064"} Oct 30 18:27:25.417005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.416946 5024 scope.go:122] "RemoveContainer" containerID="af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b" Oct 30 18:27:25.417308 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.417034 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\") pod \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " Oct 30 18:27:25.417308 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.417164 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\" (UniqueName: \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\") pod \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " Oct 30 18:27:25.417308 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.417217 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\") pod \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\" (UID: \"6861c2bd-8e3d-4e6d-98dc-5a05804aca6d\") " Oct 30 18:27:25.417308 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.417281 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca" pod "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" (UID: "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:25.417925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.417899 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.419671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.419613 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs" pod "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" (UID: "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:25.419839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.419806 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd" pod "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" (UID: "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d"). InnerVolumeSpecName "kube-api-access-8xgfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:25.419961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.419850 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls" pod "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" (UID: "6861c2bd-8e3d-4e6d-98dc-5a05804aca6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:25.452203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.452176 5024 scope.go:122] "RemoveContainer" containerID="af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b" Oct 30 18:27:25.452574 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:27:25.452549 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b\": container with ID starting with af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b not found: ID does not exist" containerID="af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b" Oct 30 18:27:25.452695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.452585 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b"} err="failed to get container status \"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b\": rpc error: code = NotFound desc = could not find container \"af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b\": container with ID starting with af202442645eef6974e20183c3ae0dc1e91ca5cf3a58d6a4f45ea07da5ddd66b not found: ID does not exist" Oct 30 18:27:25.519848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.519768 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.519848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.519798 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgfd\" (UniqueName: \"kubernetes.io/projected/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-kube-api-access-8xgfd\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.519848 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.519809 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:25.764845 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.764811 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:27:25.770013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:25.769943 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18cd5a-d87f4cb98-vb6wp"] Oct 30 18:27:27.306849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:27.306819 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" path="/var/lib/kubelet/pods/6861c2bd-8e3d-4e6d-98dc-5a05804aca6d/volumes" Oct 30 18:27:40.459124 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:27:40.459078 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:27:51.719548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:51.719509 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g"] Oct 30 18:27:51.720116 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:51.719773 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" containerID="cri-o://d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6" gracePeriod=30 Oct 30 18:27:54.372701 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:54.372665 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:27:54.373191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:54.373155 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:27:54.941738 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:54.941690 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:54.941950 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:54.941752 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:55.748895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.748861 5024 generic.go:356] "Generic (PLEG): container finished" podID="8721759e-89db-4bf6-9ec1-58874188efb3" containerID="ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41" exitCode=0 Oct 30 18:27:55.749404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.748936 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerDied","Data":"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41"} Oct 30 18:27:55.749404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.749380 5024 scope.go:122] "RemoveContainer" containerID="ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41" Oct 30 18:27:55.751128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.751101 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:55.751198 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.751169 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:55.751238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:55.751200 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:27:56.812530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:56.812485 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerStarted","Data":"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75"} Oct 30 18:27:56.989068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:56.988976 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:27:56.989244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:56.989103 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:27:56.989244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:56.989145 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:27:57.210622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.207056 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.327929 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328012 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328067 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328124 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328334 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328396 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328443 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328472 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.328537 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\") pod \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\" (UID: \"ac42e75d-0afb-418b-b322-ca68dd3fdc26\") " Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.329983 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.330183 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.330215 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.330249 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:57.330660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.330472 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:57.331639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.331569 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.331639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.331612 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.331639 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.331628 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.331828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.331642 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.331828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.331655 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.434620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.392058 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "kube-api-access-9rhtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:57.434620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.397883 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:57.434620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.398066 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:57.439317 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.436342 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.439317 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.436387 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.439317 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.436404 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhtv\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-kube-api-access-9rhtv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.455312 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.455099 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token" pod "ac42e75d-0afb-418b-b322-ca68dd3fdc26" (UID: "ac42e75d-0afb-418b-b322-ca68dd3fdc26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:57.539301 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.539251 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac42e75d-0afb-418b-b322-ca68dd3fdc26-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:27:57.844619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.841479 5024 generic.go:356] "Generic (PLEG): container finished" podID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerID="d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6" exitCode=0 Oct 30 18:27:57.844619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.842656 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" Oct 30 18:27:57.859434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.859388 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" event={"ID":"ac42e75d-0afb-418b-b322-ca68dd3fdc26","Type":"ContainerDied","Data":"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6"} Oct 30 18:27:57.860240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.860213 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g" event={"ID":"ac42e75d-0afb-418b-b322-ca68dd3fdc26","Type":"ContainerDied","Data":"5525555c1a8b52a6496120813da0ad064074551bc16185567030f6429a324b79"} Oct 30 18:27:57.860452 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.860407 5024 scope.go:122] "RemoveContainer" containerID="d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6" Oct 30 18:27:57.986990 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.986954 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:27:57.987264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:57.987233 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:27:58.032788 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.032754 5024 scope.go:122] "RemoveContainer" containerID="d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6" Oct 30 18:27:58.033428 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:27:58.033379 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6\": container with ID starting with d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6 not found: ID does not exist" containerID="d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6" Oct 30 18:27:58.033635 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.033577 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6"} err="failed to get container status \"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6\": rpc error: code = NotFound desc = could not find container \"d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6\": container with ID starting with d465fb64036037deeabda7abf22948e43166d06120a9907cd58a9f6d6be537f6 not found: ID does not exist" Oct 30 18:27:58.033844 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.033773 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g"] Oct 30 18:27:58.116624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.097504 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-hsq6g"] Oct 30 18:27:58.989038 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.988995 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:27:58.990579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:58.990547 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:27:59.654948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:59.654800 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" path="/var/lib/kubelet/pods/ac42e75d-0afb-418b-b322-ca68dd3fdc26/volumes" Oct 30 18:27:59.983483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:59.983374 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:27:59.983483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:27:59.983442 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:28:00.983494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:00.983454 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:28:00.983961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:00.983525 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:28:01.983373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:01.983333 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 18:28:01.983614 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:01.983413 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 18:28:02.987441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:02.987411 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:28:02.987959 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:02.987821 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:28:02.988437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:02.988415 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:28:06.796276 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.796234 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:06.797249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797212 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" containerName="main" Oct 30 18:28:06.797249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797241 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" containerName="main" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797265 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c1db325a-4c0b-4381-8a92-16eda65f4579" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797273 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1db325a-4c0b-4381-8a92-16eda65f4579" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797286 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797293 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797330 5024 cpu_manager.go:414] "containerMap: removing container" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797338 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerName="maas-api" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797358 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797365 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797381 5024 cpu_manager.go:414] "containerMap: removing container" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797387 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797398 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:28:06.797404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797403 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16e7315-0881-41cc-898d-378727f3335d" containerName="istio-proxy" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797582 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff46dbfd-1e57-47fc-ae2b-c5d3920cf411" containerName="main" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797620 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1db325a-4c0b-4381-8a92-16eda65f4579" containerName="maas-api" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797634 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6861c2bd-8e3d-4e6d-98dc-5a05804aca6d" containerName="maas-api" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797651 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4ddec7-9b80-404c-b783-a8d6f36f8074" containerName="maas-api" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797666 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac42e75d-0afb-418b-b322-ca68dd3fdc26" containerName="istio-proxy" Oct 30 18:28:06.797801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.797675 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabfa326-cef6-471b-b332-686a2827c2c8" containerName="istio-proxy" Oct 30 18:28:06.801783 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.801758 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.830060 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.830026 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:06.912340 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912300 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912364 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912533 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912612 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbwc\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912725 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912931 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.912985 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.912953 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:06.913080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:06.913062 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.005246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.005214 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:28:07.014579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014542 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.014771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014625 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.014771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014733 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:28:07.014771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014757 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.014982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.014982 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.014954 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015019 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015066 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015110 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015148 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbwc\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015243 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015308 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015802 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.015999 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.015976 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.017957 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.017938 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.018172 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.018156 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.039731 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.039701 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.042101 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.042079 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbwc\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.134374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.134286 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:07.458812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.458773 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:07.459913 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:28:07.459885 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cce0a0c_44b7_421c_969c_13b2e1740168.slice/crio-ea90d7f8c416c0a3913e96ef35f2db49522c3aab64451361aeeb1496de017f54 WatchSource:0}: Error finding container ea90d7f8c416c0a3913e96ef35f2db49522c3aab64451361aeeb1496de017f54: Status 404 returned error can't find the container with id ea90d7f8c416c0a3913e96ef35f2db49522c3aab64451361aeeb1496de017f54 Oct 30 18:28:07.462526 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.462496 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:07.462604 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.462580 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:07.462646 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:07.462634 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:28:08.021562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.021527 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" event={"ID":"9cce0a0c-44b7-421c-969c-13b2e1740168","Type":"ContainerStarted","Data":"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89"} Oct 30 18:28:08.021981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.021577 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" event={"ID":"9cce0a0c-44b7-421c-969c-13b2e1740168","Type":"ContainerStarted","Data":"ea90d7f8c416c0a3913e96ef35f2db49522c3aab64451361aeeb1496de017f54"} Oct 30 18:28:08.065009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.064959 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podStartSLOduration=2.064946136 podStartE2EDuration="2.064946136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:08.064762782 +0000 UTC m=+1638.885474557" watchObservedRunningTime="2026-10-30 18:28:08.064946136 +0000 UTC m=+1638.885657908" Oct 30 18:28:08.134556 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.134519 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:08.136750 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.136726 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:08.136884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:08.136768 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:09.134863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:09.134804 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:09.135302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:09.134878 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:10.135119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:10.135086 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:10.135505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:10.135138 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:10.480300 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:10.480263 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:28:10.808430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:10.808355 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:28:10.808846 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:10.808807 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" podUID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" containerName="maas-api" containerID="cri-o://5df315336e7f2d67bf3fe77299b9f84fadc42594df624524829f3b70af79c996" gracePeriod=30 Oct 30 18:28:11.173275 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.173239 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:11.183633 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.177085 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:11.212634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.210678 5024 generic.go:356] "Generic (PLEG): container finished" podID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" containerID="5df315336e7f2d67bf3fe77299b9f84fadc42594df624524829f3b70af79c996" exitCode=0 Oct 30 18:28:11.212634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.210903 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" event={"ID":"11823105-6dc6-4a83-b3fc-ccc7d9d4c740","Type":"ContainerDied","Data":"5df315336e7f2d67bf3fe77299b9f84fadc42594df624524829f3b70af79c996"} Oct 30 18:28:11.542559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.542535 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:28:11.671687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.671641 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\") pod \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " Oct 30 18:28:11.672676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.672644 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\") pod \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " Oct 30 18:28:11.673307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.673288 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\") pod \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " Oct 30 18:28:11.680651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.679729 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\" (UniqueName: \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\") pod \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\" (UID: \"11823105-6dc6-4a83-b3fc-ccc7d9d4c740\") " Oct 30 18:28:11.680651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.672320 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca" pod "11823105-6dc6-4a83-b3fc-ccc7d9d4c740" (UID: "11823105-6dc6-4a83-b3fc-ccc7d9d4c740"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:11.680651 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.679150 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs" pod "11823105-6dc6-4a83-b3fc-ccc7d9d4c740" (UID: "11823105-6dc6-4a83-b3fc-ccc7d9d4c740"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:11.681064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.681037 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls" pod "11823105-6dc6-4a83-b3fc-ccc7d9d4c740" (UID: "11823105-6dc6-4a83-b3fc-ccc7d9d4c740"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:11.697900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.696823 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph" pod "11823105-6dc6-4a83-b3fc-ccc7d9d4c740" (UID: "11823105-6dc6-4a83-b3fc-ccc7d9d4c740"). InnerVolumeSpecName "kube-api-access-vsnph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:11.781765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.781686 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:11.781765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.781718 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:11.781765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.781728 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnph\" (UniqueName: \"kubernetes.io/projected/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-kube-api-access-vsnph\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:11.781765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:11.781738 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11823105-6dc6-4a83-b3fc-ccc7d9d4c740-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:12.135480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.135389 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:12.135674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.135498 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:12.219852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.219820 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" event={"ID":"11823105-6dc6-4a83-b3fc-ccc7d9d4c740","Type":"ContainerDied","Data":"09a832a7cb1101e58c8f7c995b66aa265eda58f7e3793ba60f26b8364bd5f3b4"} Oct 30 18:28:12.220264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.219880 5024 scope.go:122] "RemoveContainer" containerID="5df315336e7f2d67bf3fe77299b9f84fadc42594df624524829f3b70af79c996" Oct 30 18:28:12.220322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.219836 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr" Oct 30 18:28:12.278861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.278812 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:28:12.289485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:12.289450 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ae3373-5fc8bd6b7d-dmbgr"] Oct 30 18:28:13.135264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:13.135229 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:13.135446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:13.135275 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:13.324400 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:13.324364 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" path="/var/lib/kubelet/pods/11823105-6dc6-4a83-b3fc-ccc7d9d4c740/volumes" Oct 30 18:28:14.135722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:14.135687 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 30 18:28:14.135953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:14.135760 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 30 18:28:15.140742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:15.140714 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:15.141266 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:15.141242 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:15.141939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:15.141919 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:19.907623 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.893713 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:28:19.917748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.917701 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:19.918011 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.917992 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:28:19.918399 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.918364 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" containerID="cri-o://8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75" gracePeriod=30 Oct 30 18:28:19.934955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.922800 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" containerID="cri-o://f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631" gracePeriod=30 Oct 30 18:28:19.935199 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:19.922941 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" containerID="cri-o://3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89" gracePeriod=30 Oct 30 18:28:22.028550 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.028499 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" containerID="cri-o://84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d" gracePeriod=45 Oct 30 18:28:22.037243 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.037204 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" containerID="cri-o://cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1" gracePeriod=45 Oct 30 18:28:22.066213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.066181 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Oct 30 18:28:22.324570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.324534 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:28:22.356398 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356373 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.356528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356421 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.356528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356471 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\" (UniqueName: \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.356528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356505 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\" (UniqueName: \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.356755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356728 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.356819 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356799 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.356882 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.356867 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\") pod \"51412cb1-9c20-42ac-8947-642002adc88d\" (UID: \"51412cb1-9c20-42ac-8947-642002adc88d\") " Oct 30 18:28:22.357465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.357432 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.357576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.357556 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.357672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.357640 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.359981 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.359940 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:22.360436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.360295 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "kube-api-access-gttfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:22.360436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.360360 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm" pod "51412cb1-9c20-42ac-8947-642002adc88d" (UID: "51412cb1-9c20-42ac-8947-642002adc88d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.376124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.376094 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:28:22.458083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458052 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458166 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458273 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458252 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458296 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\" (UniqueName: \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458335 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458342 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.458577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458413 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\") pod \"9ac176aa-6e12-4e28-803c-55c3e925ce39\" (UID: \"9ac176aa-6e12-4e28-803c-55c3e925ce39\") " Oct 30 18:28:22.458724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458665 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.458808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.458730 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.459280 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459218 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459285 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459301 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459315 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459329 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51412cb1-9c20-42ac-8947-642002adc88d-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459341 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttfc\" (UniqueName: \"kubernetes.io/projected/51412cb1-9c20-42ac-8947-642002adc88d-kube-api-access-gttfc\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459355 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51412cb1-9c20-42ac-8947-642002adc88d-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.459395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.459369 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.460874 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.460850 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "kube-api-access-kr47q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:22.461066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.461047 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.461159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.461140 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs" pod "9ac176aa-6e12-4e28-803c-55c3e925ce39" (UID: "9ac176aa-6e12-4e28-803c-55c3e925ce39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:22.560536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.560459 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ac176aa-6e12-4e28-803c-55c3e925ce39-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.560536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.560491 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr47q\" (UniqueName: \"kubernetes.io/projected/9ac176aa-6e12-4e28-803c-55c3e925ce39-kube-api-access-kr47q\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.560536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.560502 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ac176aa-6e12-4e28-803c-55c3e925ce39-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.952804 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.952771 5024 generic.go:356] "Generic (PLEG): container finished" podID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerID="cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1" exitCode=0 Oct 30 18:28:22.953003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.952820 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" Oct 30 18:28:22.953003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.952863 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" event={"ID":"9ac176aa-6e12-4e28-803c-55c3e925ce39","Type":"ContainerDied","Data":"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1"} Oct 30 18:28:22.953003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.952905 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj" event={"ID":"9ac176aa-6e12-4e28-803c-55c3e925ce39","Type":"ContainerDied","Data":"b6d28d8ac89ceb071db6106c47fb7ae8b4e5d7dc234e2c04db602a46f71f8f0b"} Oct 30 18:28:22.953003 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.952929 5024 scope.go:122] "RemoveContainer" containerID="cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1" Oct 30 18:28:22.954432 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.954407 5024 generic.go:356] "Generic (PLEG): container finished" podID="51412cb1-9c20-42ac-8947-642002adc88d" containerID="84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d" exitCode=0 Oct 30 18:28:22.954570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.954448 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" Oct 30 18:28:22.954570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.954440 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" event={"ID":"51412cb1-9c20-42ac-8947-642002adc88d","Type":"ContainerDied","Data":"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d"} Oct 30 18:28:22.954570 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.954554 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm" event={"ID":"51412cb1-9c20-42ac-8947-642002adc88d","Type":"ContainerDied","Data":"f9cbd9cc90d3a650b2e76ed2707e18f0f2f0f7dd32b80ecfbea34baa63d9ea48"} Oct 30 18:28:22.964184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.964161 5024 scope.go:122] "RemoveContainer" containerID="cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1" Oct 30 18:28:22.964971 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:22.964946 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1\": container with ID starting with cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1 not found: ID does not exist" containerID="cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1" Oct 30 18:28:22.965088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.964979 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1"} err="failed to get container status \"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1\": rpc error: code = NotFound desc = could not find container \"cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1\": container with ID starting with cc04c7a4f419fe60b4d7cb4aa177b6c7790f64ef5cc02a08f6af906f72a891e1 not found: ID does not exist" Oct 30 18:28:22.965088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.965002 5024 scope.go:122] "RemoveContainer" containerID="84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d" Oct 30 18:28:22.975229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.975202 5024 scope.go:122] "RemoveContainer" containerID="84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d" Oct 30 18:28:22.975546 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:22.975528 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d\": container with ID starting with 84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d not found: ID does not exist" containerID="84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d" Oct 30 18:28:22.975620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:22.975555 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d"} err="failed to get container status \"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d\": rpc error: code = NotFound desc = could not find container \"84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d\": container with ID starting with 84ae1a14d7bece409b1729dbaa0267b78a290eec3fbab2d8d0f5941f0305421d not found: ID does not exist" Oct 30 18:28:23.006865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.006818 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:28:23.017971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.017932 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/premium-w6-ef13cc-kserve-79bb7f864c-z8nwj"] Oct 30 18:28:23.050052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.050024 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:28:23.055094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.055065 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-w6-ef13cc-kserve-5dcbb9b48b-667fm"] Oct 30 18:28:23.311655 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.311555 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51412cb1-9c20-42ac-8947-642002adc88d" path="/var/lib/kubelet/pods/51412cb1-9c20-42ac-8947-642002adc88d/volumes" Oct 30 18:28:23.312198 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:23.312147 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" path="/var/lib/kubelet/pods/9ac176aa-6e12-4e28-803c-55c3e925ce39/volumes" Oct 30 18:28:25.416673 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.416647 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:28:25.420690 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.420663 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:28:25.424311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.424294 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:25.510135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510043 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510113 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.510376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510145 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.510376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510179 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510232 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.510376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510275 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510323 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510389 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510416 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510453 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510489 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510517 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510583 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.510674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510656 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510691 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510794 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510822 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510905 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510951 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.511049 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.510983 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\") pod \"5c8c7922-16a5-446e-b423-71341522105a\" (UID: \"5c8c7922-16a5-446e-b423-71341522105a\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511089 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511122 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\") pod \"8721759e-89db-4bf6-9ec1-58874188efb3\" (UID: \"8721759e-89db-4bf6-9ec1-58874188efb3\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511167 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511226 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511256 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.511397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.511306 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\") pod \"9cce0a0c-44b7-421c-969c-13b2e1740168\" (UID: \"9cce0a0c-44b7-421c-969c-13b2e1740168\") " Oct 30 18:28:25.512322 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.512294 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.512666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.512643 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.513821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.512909 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.513821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.513221 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.513821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.513380 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.513821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.513542 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.514108 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.513245 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.514467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514429 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.514560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514472 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.514721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514697 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.514818 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514726 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.514937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514914 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.514937 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.514927 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.515130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.515111 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.515466 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.515288 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.517724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.517695 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "kube-api-access-gtkd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.520131 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.520092 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.524877 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.524852 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.525359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.525325 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.525359 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.525329 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.525526 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.525373 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.525846 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.525812 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo" pod "8721759e-89db-4bf6-9ec1-58874188efb3" (UID: "8721759e-89db-4bf6-9ec1-58874188efb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.526093 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.526058 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "kube-api-access-lsbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.526180 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.526090 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.526411 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.526363 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.526575 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.526475 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb" pod "5c8c7922-16a5-446e-b423-71341522105a" (UID: "5c8c7922-16a5-446e-b423-71341522105a"). InnerVolumeSpecName "kube-api-access-hsxkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.528015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.527989 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token" pod "9cce0a0c-44b7-421c-969c-13b2e1740168" (UID: "9cce0a0c-44b7-421c-969c-13b2e1740168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.613719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613688 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbwc\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-kube-api-access-lsbwc\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613720 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613734 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613743 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c8c7922-16a5-446e-b423-71341522105a-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613752 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613760 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613768 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c8c7922-16a5-446e-b423-71341522105a-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613776 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613784 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613792 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkd7\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-kube-api-access-gtkd7\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613805 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613815 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613822 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613830 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613837 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8721759e-89db-4bf6-9ec1-58874188efb3-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613844 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613852 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613859 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613867 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613876 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxkb\" (UniqueName: \"kubernetes.io/projected/5c8c7922-16a5-446e-b423-71341522105a-kube-api-access-hsxkb\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613883 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cce0a0c-44b7-421c-969c-13b2e1740168-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613892 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8721759e-89db-4bf6-9ec1-58874188efb3-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613905 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8721759e-89db-4bf6-9ec1-58874188efb3-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613917 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c8c7922-16a5-446e-b423-71341522105a-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613929 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cce0a0c-44b7-421c-969c-13b2e1740168-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613942 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8721759e-89db-4bf6-9ec1-58874188efb3-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.613988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:25.613952 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cce0a0c-44b7-421c-969c-13b2e1740168-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:26.002925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.002887 5024 generic.go:356] "Generic (PLEG): container finished" podID="8721759e-89db-4bf6-9ec1-58874188efb3" containerID="8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75" exitCode=0 Oct 30 18:28:26.003343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.003316 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerDied","Data":"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75"} Oct 30 18:28:26.003508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.003492 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" event={"ID":"8721759e-89db-4bf6-9ec1-58874188efb3","Type":"ContainerDied","Data":"7a51971e9b927d0a1c2338eefb75bd6dd1695a97cbc57b85b071bf5ff24813f2"} Oct 30 18:28:26.003661 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.003646 5024 scope.go:122] "RemoveContainer" containerID="8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75" Oct 30 18:28:26.003939 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.003921 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d" Oct 30 18:28:26.077924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.052977 5024 generic.go:356] "Generic (PLEG): container finished" podID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerID="3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89" exitCode=0 Oct 30 18:28:26.077924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.053153 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" event={"ID":"9cce0a0c-44b7-421c-969c-13b2e1740168","Type":"ContainerDied","Data":"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89"} Oct 30 18:28:26.077924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.053260 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" event={"ID":"9cce0a0c-44b7-421c-969c-13b2e1740168","Type":"ContainerDied","Data":"ea90d7f8c416c0a3913e96ef35f2db49522c3aab64451361aeeb1496de017f54"} Oct 30 18:28:26.077924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.053375 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb" Oct 30 18:28:26.154068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.150864 5024 scope.go:122] "RemoveContainer" containerID="ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41" Oct 30 18:28:26.154068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.152131 5024 generic.go:356] "Generic (PLEG): container finished" podID="5c8c7922-16a5-446e-b423-71341522105a" containerID="f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631" exitCode=0 Oct 30 18:28:26.154068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.152189 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" event={"ID":"5c8c7922-16a5-446e-b423-71341522105a","Type":"ContainerDied","Data":"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631"} Oct 30 18:28:26.154068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.152227 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" event={"ID":"5c8c7922-16a5-446e-b423-71341522105a","Type":"ContainerDied","Data":"2b7a799cd5ce96e9ed9280236eb9ebcae3c2fae25d473555c7600c33d8d42bc6"} Oct 30 18:28:26.154068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.152294 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr" Oct 30 18:28:26.175094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.174947 5024 scope.go:122] "RemoveContainer" containerID="8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75" Oct 30 18:28:26.175420 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:26.175389 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75\": container with ID starting with 8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75 not found: ID does not exist" containerID="8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75" Oct 30 18:28:26.175528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.175433 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75"} err="failed to get container status \"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75\": rpc error: code = NotFound desc = could not find container \"8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75\": container with ID starting with 8b93d33c5b25fff5be14353581cb04cb6ba708c0dccdbc916bf7e3c9e1608a75 not found: ID does not exist" Oct 30 18:28:26.175528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.175453 5024 scope.go:122] "RemoveContainer" containerID="ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41" Oct 30 18:28:26.175820 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:26.175798 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41\": container with ID starting with ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41 not found: ID does not exist" containerID="ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41" Oct 30 18:28:26.175902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.175838 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41"} err="failed to get container status \"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41\": rpc error: code = NotFound desc = could not find container \"ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41\": container with ID starting with ccc34fd6b039b4f728e8955eaf6d80b275b6898831db47165dcc9f49662dcb41 not found: ID does not exist" Oct 30 18:28:26.175902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.175858 5024 scope.go:122] "RemoveContainer" containerID="3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89" Oct 30 18:28:26.187349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.187319 5024 scope.go:122] "RemoveContainer" containerID="3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89" Oct 30 18:28:26.187796 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:26.187734 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89\": container with ID starting with 3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89 not found: ID does not exist" containerID="3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89" Oct 30 18:28:26.187796 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.187773 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89"} err="failed to get container status \"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89\": rpc error: code = NotFound desc = could not find container \"3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89\": container with ID starting with 3fe09ec4e18bd7a40b610a2b790941d6f95c011599d66dbba13a68dfe44a9a89 not found: ID does not exist" Oct 30 18:28:26.187940 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.187797 5024 scope.go:122] "RemoveContainer" containerID="f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631" Oct 30 18:28:26.200487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.200454 5024 scope.go:122] "RemoveContainer" containerID="f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631" Oct 30 18:28:26.200857 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:26.200828 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631\": container with ID starting with f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631 not found: ID does not exist" containerID="f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631" Oct 30 18:28:26.200974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.200864 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631"} err="failed to get container status \"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631\": rpc error: code = NotFound desc = could not find container \"f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631\": container with ID starting with f10a46723629144d4ec8ff23533fb190a9aa9a159aa676069b57070ba6fc7631 not found: ID does not exist" Oct 30 18:28:26.214381 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.214351 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:28:26.219147 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.219122 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h45tr"] Oct 30 18:28:26.250471 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.250367 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:26.255302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.255275 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cb7zb"] Oct 30 18:28:26.292463 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.292426 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:28:26.300400 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:26.300374 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-cfn5d"] Oct 30 18:28:27.307841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:27.307807 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8c7922-16a5-446e-b423-71341522105a" path="/var/lib/kubelet/pods/5c8c7922-16a5-446e-b423-71341522105a/volumes" Oct 30 18:28:27.308368 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:27.308352 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" path="/var/lib/kubelet/pods/8721759e-89db-4bf6-9ec1-58874188efb3/volumes" Oct 30 18:28:27.308834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:27.308820 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" path="/var/lib/kubelet/pods/9cce0a0c-44b7-421c-969c-13b2e1740168/volumes" Oct 30 18:28:30.536028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:30.535922 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:28:33.832095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:33.832060 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:28:33.832573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:33.832276 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" podUID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" containerName="maas-api" containerID="cri-o://329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1" gracePeriod=30 Oct 30 18:28:34.241165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.240897 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:28:34.350066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.349874 5024 generic.go:356] "Generic (PLEG): container finished" podID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" containerID="329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1" exitCode=0 Oct 30 18:28:34.350066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.349945 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" event={"ID":"311c06ef-f28e-4539-9d73-0d0e51b80a2d","Type":"ContainerDied","Data":"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1"} Oct 30 18:28:34.350066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.350003 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" event={"ID":"311c06ef-f28e-4539-9d73-0d0e51b80a2d","Type":"ContainerDied","Data":"c98b7d74b23c18cce5533248e1d84d1b0ec73082cff0476a497621ea278bf5f1"} Oct 30 18:28:34.350066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.350028 5024 scope.go:122] "RemoveContainer" containerID="329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1" Oct 30 18:28:34.350443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.350155 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn" Oct 30 18:28:34.373996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.373973 5024 scope.go:122] "RemoveContainer" containerID="329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1" Oct 30 18:28:34.377821 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:34.375268 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1\": container with ID starting with 329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1 not found: ID does not exist" containerID="329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1" Oct 30 18:28:34.377821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.375312 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1"} err="failed to get container status \"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1\": rpc error: code = NotFound desc = could not find container \"329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1\": container with ID starting with 329152c6cc328e80bda67d1e45f2ef3112ab01ac64b06de2c65787e19c1a30d1 not found: ID does not exist" Oct 30 18:28:34.393019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.391210 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\") pod \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " Oct 30 18:28:34.393019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.391341 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\") pod \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " Oct 30 18:28:34.393019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.391514 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\") pod \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " Oct 30 18:28:34.393019 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.391570 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\" (UniqueName: \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\") pod \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\" (UID: \"311c06ef-f28e-4539-9d73-0d0e51b80a2d\") " Oct 30 18:28:34.394532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.394490 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca" pod "311c06ef-f28e-4539-9d73-0d0e51b80a2d" (UID: "311c06ef-f28e-4539-9d73-0d0e51b80a2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:34.397403 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.397379 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls" pod "311c06ef-f28e-4539-9d73-0d0e51b80a2d" (UID: "311c06ef-f28e-4539-9d73-0d0e51b80a2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:34.407119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.404758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs" pod "311c06ef-f28e-4539-9d73-0d0e51b80a2d" (UID: "311c06ef-f28e-4539-9d73-0d0e51b80a2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:34.407119 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.405059 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z" pod "311c06ef-f28e-4539-9d73-0d0e51b80a2d" (UID: "311c06ef-f28e-4539-9d73-0d0e51b80a2d"). InnerVolumeSpecName "kube-api-access-vh52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:34.494424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.494339 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311c06ef-f28e-4539-9d73-0d0e51b80a2d-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.494424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.494372 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.494424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.494384 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311c06ef-f28e-4539-9d73-0d0e51b80a2d-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.494424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.494401 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh52z\" (UniqueName: \"kubernetes.io/projected/311c06ef-f28e-4539-9d73-0d0e51b80a2d-kube-api-access-vh52z\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:34.699889 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.699845 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:28:34.705256 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:34.705216 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e5d168-74f875566b-75vwn"] Oct 30 18:28:35.306824 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.306774 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" path="/var/lib/kubelet/pods/311c06ef-f28e-4539-9d73-0d0e51b80a2d/volumes" Oct 30 18:28:35.536707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.536661 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" podUID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" containerName="payload-pre-processing" containerID="cri-o://a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2" gracePeriod=25 Oct 30 18:28:35.825421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.825394 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:28:35.911345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.911232 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\" (UniqueName: \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\") pod \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " Oct 30 18:28:35.911345 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.911320 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\") pod \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\" (UID: \"0fd90bf0-2165-4e1b-9d1f-76b616c036ed\") " Oct 30 18:28:35.911828 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.911801 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume" pod "0fd90bf0-2165-4e1b-9d1f-76b616c036ed" (UID: "0fd90bf0-2165-4e1b-9d1f-76b616c036ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:35.912023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.912003 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-plugins-config-volume\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.913773 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:35.913746 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d" pod "0fd90bf0-2165-4e1b-9d1f-76b616c036ed" (UID: "0fd90bf0-2165-4e1b-9d1f-76b616c036ed"). InnerVolumeSpecName "kube-api-access-x8q8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:36.013188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.013145 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8q8d\" (UniqueName: \"kubernetes.io/projected/0fd90bf0-2165-4e1b-9d1f-76b616c036ed-kube-api-access-x8q8d\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:36.366429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.366340 5024 generic.go:356] "Generic (PLEG): container finished" podID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" containerID="a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2" exitCode=0 Oct 30 18:28:36.366429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.366388 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" Oct 30 18:28:36.366932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.366430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" event={"ID":"0fd90bf0-2165-4e1b-9d1f-76b616c036ed","Type":"ContainerDied","Data":"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2"} Oct 30 18:28:36.366932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.366485 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964" event={"ID":"0fd90bf0-2165-4e1b-9d1f-76b616c036ed","Type":"ContainerDied","Data":"2f5448085c2a8c202cb2fcfef10047bef45b999ae062d6fab638c46640c558c0"} Oct 30 18:28:36.366932 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.366514 5024 scope.go:122] "RemoveContainer" containerID="a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2" Oct 30 18:28:36.377452 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.377433 5024 scope.go:122] "RemoveContainer" containerID="a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2" Oct 30 18:28:36.377802 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:36.377781 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2\": container with ID starting with a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2 not found: ID does not exist" containerID="a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2" Oct 30 18:28:36.377868 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.377812 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2"} err="failed to get container status \"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2\": rpc error: code = NotFound desc = could not find container \"a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2\": container with ID starting with a685a115f4f3fbaed362a2d674303c7f4b37adef41942f380f9a20f20e9615e2 not found: ID does not exist" Oct 30 18:28:36.427648 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.427573 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:28:36.434271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:36.434237 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e5d168-656d868fd7-hk964"] Oct 30 18:28:37.305548 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:37.305517 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" path="/var/lib/kubelet/pods/0fd90bf0-2165-4e1b-9d1f-76b616c036ed/volumes" Oct 30 18:28:40.459127 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:40.459094 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:28:42.386023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.385979 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:28:42.386916 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.386840 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" containerID="cri-o://c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f" gracePeriod=30 Oct 30 18:28:42.397513 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.397476 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl"] Oct 30 18:28:42.397794 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.397760 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" containerID="cri-o://ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627" gracePeriod=30 Oct 30 18:28:42.412262 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.412230 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d"] Oct 30 18:28:42.412517 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:42.412476 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" containerID="cri-o://279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375" gracePeriod=30 Oct 30 18:28:47.751306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.751274 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820339 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820395 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820446 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820481 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820511 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820615 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820661 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820707 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.820863 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\") pod \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\" (UID: \"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54\") " Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.821243 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.821558 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.822010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.821744 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.823543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.823134 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.823543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.823335 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.823543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.823505 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.826106 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.826078 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.826253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.826117 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "kube-api-access-fn5zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.827563 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.827541 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.828026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.827841 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy" pod "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" (UID: "0c9ae01f-ab17-43a5-841d-d3e7c03e2d54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.830418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.830396 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:28:47.834034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.834014 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:28:47.905197 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.905164 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922044 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922107 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922137 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922364 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922479 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922537 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922576 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922626 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922657 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922691 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922722 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922749 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922775 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922858 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922928 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.922991 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923020 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\") pod \"38aea5e9-5448-494c-b350-1867c4c56275\" (UID: \"38aea5e9-5448-494c-b350-1867c4c56275\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923055 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\") pod \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\" (UID: \"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1\") " Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923734 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923757 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923770 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923781 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923795 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5zj\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-kube-api-access-fn5zj\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923808 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923822 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.924065 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.923835 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.925401 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.924709 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.925401 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.924903 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.925401 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.925350 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.925791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.925763 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.925901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.925878 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.925998 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.925969 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.926446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.926273 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.926536 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.926515 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.926905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.926739 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.927076 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.927050 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.930583 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.930537 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.930863 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.930823 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "kube-api-access-k5sqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.931812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.931788 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.932202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.932174 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token" pod "38aea5e9-5448-494c-b350-1867c4c56275" (UID: "38aea5e9-5448-494c-b350-1867c4c56275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.932574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.932548 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.933270 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.932750 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.933827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.933549 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "kube-api-access-9znnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.934044 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:47.934019 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo" pod "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" (UID: "d5dc2857-2d5f-4981-a5b9-bee0ed997dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:48.025388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025343 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025379 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38aea5e9-5448-494c-b350-1867c4c56275-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025392 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025404 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025417 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38aea5e9-5448-494c-b350-1867c4c56275-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025430 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025441 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025452 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025463 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025473 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025484 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025494 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025506 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025518 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025530 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5sqg\" (UniqueName: \"kubernetes.io/projected/38aea5e9-5448-494c-b350-1867c4c56275-kube-api-access-k5sqg\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025544 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025560 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znnd\" (UniqueName: \"kubernetes.io/projected/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1-kube-api-access-9znnd\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.025717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.025571 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38aea5e9-5448-494c-b350-1867c4c56275-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.461267 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.461229 5024 generic.go:356] "Generic (PLEG): container finished" podID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerID="c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f" exitCode=0 Oct 30 18:28:48.461443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.461355 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" event={"ID":"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54","Type":"ContainerDied","Data":"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f"} Oct 30 18:28:48.461443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.461407 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" event={"ID":"0c9ae01f-ab17-43a5-841d-d3e7c03e2d54","Type":"ContainerDied","Data":"9d54320f0c40950b6a6ec880ca32c0b65f4687b9126cac1d7416c1eaaff7a4ea"} Oct 30 18:28:48.461443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.461435 5024 scope.go:122] "RemoveContainer" containerID="c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f" Oct 30 18:28:48.461654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.461590 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw" Oct 30 18:28:48.471074 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.471039 5024 generic.go:356] "Generic (PLEG): container finished" podID="38aea5e9-5448-494c-b350-1867c4c56275" containerID="279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375" exitCode=0 Oct 30 18:28:48.471242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.471187 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" event={"ID":"38aea5e9-5448-494c-b350-1867c4c56275","Type":"ContainerDied","Data":"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375"} Oct 30 18:28:48.471314 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.471239 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" event={"ID":"38aea5e9-5448-494c-b350-1867c4c56275","Type":"ContainerDied","Data":"4580723a901538ddebb10d9c4935ff3ad0ba29afc5747f77d8fba9f1504f9b9f"} Oct 30 18:28:48.471369 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.471329 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d" Oct 30 18:28:48.474145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.474107 5024 generic.go:356] "Generic (PLEG): container finished" podID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerID="ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627" exitCode=0 Oct 30 18:28:48.478111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.478079 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" event={"ID":"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1","Type":"ContainerDied","Data":"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627"} Oct 30 18:28:48.478253 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.478152 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" event={"ID":"d5dc2857-2d5f-4981-a5b9-bee0ed997dc1","Type":"ContainerDied","Data":"2ae5a945091a0b91b783115afe180ee9b65e48d8532540e3624023c52be5162e"} Oct 30 18:28:48.478320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.478273 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl" Oct 30 18:28:48.478895 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.478798 5024 scope.go:122] "RemoveContainer" containerID="c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f" Oct 30 18:28:48.479181 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:48.479142 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f\": container with ID starting with c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f not found: ID does not exist" containerID="c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f" Oct 30 18:28:48.479302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.479184 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f"} err="failed to get container status \"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f\": rpc error: code = NotFound desc = could not find container \"c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f\": container with ID starting with c82d75948f24cf584dff9c88f9abdce189b17dd19ccaf79b3b6ac2d2e3368b4f not found: ID does not exist" Oct 30 18:28:48.479302 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.479204 5024 scope.go:122] "RemoveContainer" containerID="279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375" Oct 30 18:28:48.497724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.497698 5024 scope.go:122] "RemoveContainer" containerID="279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375" Oct 30 18:28:48.500393 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:48.500365 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375\": container with ID starting with 279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375 not found: ID does not exist" containerID="279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375" Oct 30 18:28:48.500494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.500399 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375"} err="failed to get container status \"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375\": rpc error: code = NotFound desc = could not find container \"279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375\": container with ID starting with 279bd7e22f8fd6e38d3e794c8a435c109a7bc9bbc8d5fce1cea703c08490e375 not found: ID does not exist" Oct 30 18:28:48.500494 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.500418 5024 scope.go:122] "RemoveContainer" containerID="ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627" Oct 30 18:28:48.510530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.510511 5024 scope.go:122] "RemoveContainer" containerID="ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627" Oct 30 18:28:48.510907 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:48.510886 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627\": container with ID starting with ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627 not found: ID does not exist" containerID="ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627" Oct 30 18:28:48.510988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.510916 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627"} err="failed to get container status \"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627\": rpc error: code = NotFound desc = could not find container \"ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627\": container with ID starting with ed32bd60407af5ce615a621330fcc68287f336ac71a940ff454d162892f7f627 not found: ID does not exist" Oct 30 18:28:48.530334 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.530301 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:28:48.541337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.541304 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-bznzw"] Oct 30 18:28:48.600694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.600663 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl"] Oct 30 18:28:48.605948 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.605915 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-99cpl"] Oct 30 18:28:48.635493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.635465 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d"] Oct 30 18:28:48.642576 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:48.642540 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-clt6d"] Oct 30 18:28:49.313274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:49.313234 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" path="/var/lib/kubelet/pods/0c9ae01f-ab17-43a5-841d-d3e7c03e2d54/volumes" Oct 30 18:28:49.314015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:49.313993 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38aea5e9-5448-494c-b350-1867c4c56275" path="/var/lib/kubelet/pods/38aea5e9-5448-494c-b350-1867c4c56275/volumes" Oct 30 18:28:49.314674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:49.314653 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" path="/var/lib/kubelet/pods/d5dc2857-2d5f-4981-a5b9-bee0ed997dc1/volumes" Oct 30 18:28:53.132873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.132834 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:28:53.133383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.133101 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" podUID="0580522a-0b43-402b-838a-1896f1ffca93" containerName="maas-api" containerID="cri-o://cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0" gracePeriod=30 Oct 30 18:28:53.556660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.554039 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:28:53.561945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.561892 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\") pod \"0580522a-0b43-402b-838a-1896f1ffca93\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " Oct 30 18:28:53.562141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.562044 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\" (UniqueName: \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\") pod \"0580522a-0b43-402b-838a-1896f1ffca93\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " Oct 30 18:28:53.562141 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.562078 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\") pod \"0580522a-0b43-402b-838a-1896f1ffca93\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " Oct 30 18:28:53.562267 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.562171 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") pod \"0580522a-0b43-402b-838a-1896f1ffca93\" (UID: \"0580522a-0b43-402b-838a-1896f1ffca93\") " Oct 30 18:28:53.563730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.563696 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca" pod "0580522a-0b43-402b-838a-1896f1ffca93" (UID: "0580522a-0b43-402b-838a-1896f1ffca93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:53.565624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.565258 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc" pod "0580522a-0b43-402b-838a-1896f1ffca93" (UID: "0580522a-0b43-402b-838a-1896f1ffca93"). InnerVolumeSpecName "kube-api-access-jsltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:53.567777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.567733 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs" pod "0580522a-0b43-402b-838a-1896f1ffca93" (UID: "0580522a-0b43-402b-838a-1896f1ffca93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:53.580229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.580193 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls" pod "0580522a-0b43-402b-838a-1896f1ffca93" (UID: "0580522a-0b43-402b-838a-1896f1ffca93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:53.664624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.663972 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:53.664624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.664015 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0580522a-0b43-402b-838a-1896f1ffca93-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:53.664624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.664029 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsltc\" (UniqueName: \"kubernetes.io/projected/0580522a-0b43-402b-838a-1896f1ffca93-kube-api-access-jsltc\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:53.664624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.664042 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0580522a-0b43-402b-838a-1896f1ffca93-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:28:53.676620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.665901 5024 generic.go:356] "Generic (PLEG): container finished" podID="0580522a-0b43-402b-838a-1896f1ffca93" containerID="cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0" exitCode=0 Oct 30 18:28:53.676620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.665991 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" event={"ID":"0580522a-0b43-402b-838a-1896f1ffca93","Type":"ContainerDied","Data":"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0"} Oct 30 18:28:53.676620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.666046 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" event={"ID":"0580522a-0b43-402b-838a-1896f1ffca93","Type":"ContainerDied","Data":"b853ef01a3fb0b7b158a67f73e6bccafc3ca82d1beb81ad82773ab3c76fdbde3"} Oct 30 18:28:53.676620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.666077 5024 scope.go:122] "RemoveContainer" containerID="cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0" Oct 30 18:28:53.676620 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.666363 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g" Oct 30 18:28:53.756426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.756355 5024 scope.go:122] "RemoveContainer" containerID="cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0" Oct 30 18:28:53.756821 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:28:53.756788 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0\": container with ID starting with cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0 not found: ID does not exist" containerID="cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0" Oct 30 18:28:53.756924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.756824 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0"} err="failed to get container status \"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0\": rpc error: code = NotFound desc = could not find container \"cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0\": container with ID starting with cce88272dd7682ab8f21f437b08d14a94af5bdc9a920254e882fafa312c505c0 not found: ID does not exist" Oct 30 18:28:53.788857 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.788773 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:28:53.794524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:53.794470 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b2b596-5f4c6dd46f-gzj2g"] Oct 30 18:28:55.305891 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:55.305860 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0580522a-0b43-402b-838a-1896f1ffca93" path="/var/lib/kubelet/pods/0580522a-0b43-402b-838a-1896f1ffca93/volumes" Oct 30 18:28:59.560716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:59.560680 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:28:59.574911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:59.574868 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" podUID="94164130-02ad-4a1c-8280-db911430ad97" containerName="maas-api" containerID="cri-o://1efb4dfe620da7933777b6b3d43690595bfcace4b38f3ce98b5cdebe86699e4b" gracePeriod=30 Oct 30 18:28:59.711573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:59.711522 5024 generic.go:356] "Generic (PLEG): container finished" podID="94164130-02ad-4a1c-8280-db911430ad97" containerID="1efb4dfe620da7933777b6b3d43690595bfcace4b38f3ce98b5cdebe86699e4b" exitCode=0 Oct 30 18:28:59.711785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:59.711632 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" event={"ID":"94164130-02ad-4a1c-8280-db911430ad97","Type":"ContainerDied","Data":"1efb4dfe620da7933777b6b3d43690595bfcace4b38f3ce98b5cdebe86699e4b"} Oct 30 18:28:59.728474 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:28:59.728433 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" podUID="94164130-02ad-4a1c-8280-db911430ad97" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.57:8443/health\": dial tcp 10.132.0.57:8443: connect: connection refused" Oct 30 18:29:00.088888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.088856 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:29:00.207865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.207832 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\") pod \"94164130-02ad-4a1c-8280-db911430ad97\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " Oct 30 18:29:00.207865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.207878 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\" (UniqueName: \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\") pod \"94164130-02ad-4a1c-8280-db911430ad97\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " Oct 30 18:29:00.208150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.207902 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\") pod \"94164130-02ad-4a1c-8280-db911430ad97\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " Oct 30 18:29:00.208150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.208037 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\") pod \"94164130-02ad-4a1c-8280-db911430ad97\" (UID: \"94164130-02ad-4a1c-8280-db911430ad97\") " Oct 30 18:29:00.208335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.208306 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca" pod "94164130-02ad-4a1c-8280-db911430ad97" (UID: "94164130-02ad-4a1c-8280-db911430ad97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:00.209203 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.209174 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94164130-02ad-4a1c-8280-db911430ad97-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:00.210459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.210430 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8" pod "94164130-02ad-4a1c-8280-db911430ad97" (UID: "94164130-02ad-4a1c-8280-db911430ad97"). InnerVolumeSpecName "kube-api-access-g6tw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:00.210547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.210492 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls" pod "94164130-02ad-4a1c-8280-db911430ad97" (UID: "94164130-02ad-4a1c-8280-db911430ad97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:00.210547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.210509 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs" pod "94164130-02ad-4a1c-8280-db911430ad97" (UID: "94164130-02ad-4a1c-8280-db911430ad97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:00.311105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.311026 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6tw8\" (UniqueName: \"kubernetes.io/projected/94164130-02ad-4a1c-8280-db911430ad97-kube-api-access-g6tw8\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:00.311105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.311057 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:00.311105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.311069 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94164130-02ad-4a1c-8280-db911430ad97-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:00.719670 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.719641 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" Oct 30 18:29:00.720184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.719640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht" event={"ID":"94164130-02ad-4a1c-8280-db911430ad97","Type":"ContainerDied","Data":"42a047253bd4ba6b485b47220407ec25b0385f1917ef8bdadf155c8d0e14cb9f"} Oct 30 18:29:00.720184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.719776 5024 scope.go:122] "RemoveContainer" containerID="1efb4dfe620da7933777b6b3d43690595bfcace4b38f3ce98b5cdebe86699e4b" Oct 30 18:29:00.769091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.769058 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:29:00.779720 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:00.779688 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef13cc-69fb588fcd-gjlht"] Oct 30 18:29:01.305709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:01.305670 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94164130-02ad-4a1c-8280-db911430ad97" path="/var/lib/kubelet/pods/94164130-02ad-4a1c-8280-db911430ad97/volumes" Oct 30 18:29:02.071451 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:02.071418 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:29:02.072037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:02.071641 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" containerID="cri-o://48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3" gracePeriod=30 Oct 30 18:29:02.933094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:02.933048 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" containerID="cri-o://a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8" gracePeriod=45 Oct 30 18:29:03.232995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.232969 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.347645 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.347763 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.347797 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.347889 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\" (UniqueName: \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.347930 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348010 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\") pod \"8ecb2f63-a597-4615-b3c8-60add2150149\" (UID: \"8ecb2f63-a597-4615-b3c8-60add2150149\") " Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348045 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348199 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348312 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348693 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348712 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.348769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.348726 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.353284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.353238 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:03.369565 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.369156 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "kube-api-access-r6v6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:03.384659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.384573 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm" pod "8ecb2f63-a597-4615-b3c8-60add2150149" (UID: "8ecb2f63-a597-4615-b3c8-60add2150149"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.455166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.455078 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6v6v\" (UniqueName: \"kubernetes.io/projected/8ecb2f63-a597-4615-b3c8-60add2150149-kube-api-access-r6v6v\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.455166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.455115 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ecb2f63-a597-4615-b3c8-60add2150149-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.455166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.455130 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ecb2f63-a597-4615-b3c8-60add2150149-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.740374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.740287 5024 generic.go:356] "Generic (PLEG): container finished" podID="8ecb2f63-a597-4615-b3c8-60add2150149" containerID="a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8" exitCode=0 Oct 30 18:29:03.740374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.740331 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" Oct 30 18:29:03.740686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.740377 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" event={"ID":"8ecb2f63-a597-4615-b3c8-60add2150149","Type":"ContainerDied","Data":"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8"} Oct 30 18:29:03.740686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.740424 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc" event={"ID":"8ecb2f63-a597-4615-b3c8-60add2150149","Type":"ContainerDied","Data":"abcd2201d13a4f28a5d02a479e4356355524e532e55ebeed4226caeb23bbb138"} Oct 30 18:29:03.740686 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.740447 5024 scope.go:122] "RemoveContainer" containerID="a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8" Oct 30 18:29:03.762353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.762308 5024 scope.go:122] "RemoveContainer" containerID="a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8" Oct 30 18:29:03.762879 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:03.762847 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8\": container with ID starting with a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8 not found: ID does not exist" containerID="a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8" Oct 30 18:29:03.762996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.762895 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8"} err="failed to get container status \"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8\": rpc error: code = NotFound desc = could not find container \"a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8\": container with ID starting with a18cdc4a38e38dd8e7764842c43e12088fae23b4253ffa7b0cfa7e597dd6bfe8 not found: ID does not exist" Oct 30 18:29:03.795858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.795820 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:29:03.806212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:03.806180 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-w0-6ce5eb-kserve-6c9c7c965f-pj9bc"] Oct 30 18:29:04.248295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:04.248259 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:29:04.248724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:04.248330 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:29:05.306086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:05.306048 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" path="/var/lib/kubelet/pods/8ecb2f63-a597-4615-b3c8-60add2150149/volumes" Oct 30 18:29:07.352067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.352001 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:29:07.445886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.445845 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.445951 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446029 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446084 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446124 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446173 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446216 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446250 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446349 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446327 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\") pod \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\" (UID: \"11eb5e2b-31d5-46a4-8482-09fc9691a50c\") " Oct 30 18:29:07.446501 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446433 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.446568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446494 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.446568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.446515 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.447063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.447014 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:07.447063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.447052 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.447288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.447147 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.447288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.447170 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.447288 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.447186 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.448699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.448670 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "kube-api-access-b9xhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.449386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.449356 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.449386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.449373 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.449560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.449541 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo" pod "11eb5e2b-31d5-46a4-8482-09fc9691a50c" (UID: "11eb5e2b-31d5-46a4-8482-09fc9691a50c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:07.548505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548473 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.548505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548504 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xhk\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-kube-api-access-b9xhk\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.548505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548514 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.548825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548523 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.548825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548532 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.548825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.548542 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11eb5e2b-31d5-46a4-8482-09fc9691a50c-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.791268 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.791236 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:29:07.791842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.791807 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" containerID="cri-o://d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b" gracePeriod=30 Oct 30 18:29:07.799840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.799738 5024 generic.go:356] "Generic (PLEG): container finished" podID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerID="48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3" exitCode=0 Oct 30 18:29:07.800008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.799860 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" event={"ID":"11eb5e2b-31d5-46a4-8482-09fc9691a50c","Type":"ContainerDied","Data":"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3"} Oct 30 18:29:07.800008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.799914 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" event={"ID":"11eb5e2b-31d5-46a4-8482-09fc9691a50c","Type":"ContainerDied","Data":"57aa6cf7bf0e6a767da6e407599c97df08a07c29abcd1577ad0b44eb52f32623"} Oct 30 18:29:07.800008 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.799943 5024 scope.go:122] "RemoveContainer" containerID="48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3" Oct 30 18:29:07.800164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.800069 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c" Oct 30 18:29:07.811755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.811543 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:29:07.811918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.811808 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" containerID="cri-o://da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6" gracePeriod=30 Oct 30 18:29:07.822083 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.822053 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:29:07.822326 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.822274 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" containerID="cri-o://c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5" gracePeriod=30 Oct 30 18:29:07.836544 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.836520 5024 scope.go:122] "RemoveContainer" containerID="48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3" Oct 30 18:29:07.836968 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:07.836933 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3\": container with ID starting with 48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3 not found: ID does not exist" containerID="48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3" Oct 30 18:29:07.837114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.836976 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3"} err="failed to get container status \"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3\": rpc error: code = NotFound desc = could not find container \"48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3\": container with ID starting with 48862badce3f4727a8c14d7d9cae544f0356a5a5df697abaadf75299b376d0e3 not found: ID does not exist" Oct 30 18:29:07.870840 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.870809 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:29:07.874975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:07.874942 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vjf4c"] Oct 30 18:29:09.316165 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:09.316123 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" path="/var/lib/kubelet/pods/11eb5e2b-31d5-46a4-8482-09fc9691a50c/volumes" Oct 30 18:29:10.457060 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:10.457021 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:29:12.746934 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.746898 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:29:12.747472 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747457 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747476 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747497 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747502 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747510 5024 cpu_manager.go:414] "containerMap: removing container" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747515 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747528 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0580522a-0b43-402b-838a-1896f1ffca93" containerName="maas-api" Oct 30 18:29:12.747537 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747533 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0580522a-0b43-402b-838a-1896f1ffca93" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747542 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747550 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747561 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747566 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747573 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747578 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747585 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747592 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747625 5024 cpu_manager.go:414] "containerMap: removing container" podUID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747631 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747644 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747648 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747655 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747659 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747666 5024 cpu_manager.go:414] "containerMap: removing container" podUID="94164130-02ad-4a1c-8280-db911430ad97" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747670 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94164130-02ad-4a1c-8280-db911430ad97" containerName="maas-api" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747676 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" containerName="payload-pre-processing" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747683 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" containerName="payload-pre-processing" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747694 5024 cpu_manager.go:414] "containerMap: removing container" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747699 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747710 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747714 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747721 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" Oct 30 18:29:12.747762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747725 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747824 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="51412cb1-9c20-42ac-8947-642002adc88d" containerName="main" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747833 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecb2f63-a597-4615-b3c8-60add2150149" containerName="main" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747844 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9ae01f-ab17-43a5-841d-d3e7c03e2d54" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747852 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11eb5e2b-31d5-46a4-8482-09fc9691a50c" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747859 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cce0a0c-44b7-421c-969c-13b2e1740168" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747864 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="38aea5e9-5448-494c-b350-1867c4c56275" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747871 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0580522a-0b43-402b-838a-1896f1ffca93" containerName="maas-api" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747876 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747880 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac176aa-6e12-4e28-803c-55c3e925ce39" containerName="main" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747885 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8c7922-16a5-446e-b423-71341522105a" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747890 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5dc2857-2d5f-4981-a5b9-bee0ed997dc1" containerName="istio-proxy" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747896 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="311c06ef-f28e-4539-9d73-0d0e51b80a2d" containerName="maas-api" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747903 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11823105-6dc6-4a83-b3fc-ccc7d9d4c740" containerName="maas-api" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747909 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="94164130-02ad-4a1c-8280-db911430ad97" containerName="maas-api" Oct 30 18:29:12.748461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.747915 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd90bf0-2165-4e1b-9d1f-76b616c036ed" containerName="payload-pre-processing" Oct 30 18:29:12.752933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.752904 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.775436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.770902 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:29:12.775436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.772637 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69c44c\"/\"kube-root-ca.crt\"" Oct 30 18:29:12.775436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.773680 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69c44c\"/\"openshift-service-ca.crt\"" Oct 30 18:29:12.778995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.778951 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69c44c\"/\"default-dockercfg-mv76f\"" Oct 30 18:29:12.779170 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.779086 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69c44c\"/\"test-model-69c44c-kserve-self-signed-certs\"" Oct 30 18:29:12.811080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811043 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.811080 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811086 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.811376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811111 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qsq\" (UniqueName: \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.811376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811282 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.811376 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811328 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.811588 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.811563 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.912924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.912884 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.913134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.912939 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.913134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.912964 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qsq\" (UniqueName: \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.913134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913012 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.913134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913046 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.913343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.916640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913652 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.916640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913757 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.916640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.913766 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.916640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.915891 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.916640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.916050 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:12.932925 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:12.932888 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qsq\" (UniqueName: \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\") pod \"test-model-69c44c-kserve-6c64877c97-6jdvn\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:13.073401 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.072876 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:13.198522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.198151 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219097 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219216 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219253 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219354 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219387 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219457 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219520 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219662 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\") pod \"3996e964-79db-4b61-b069-96e161ad78b9\" (UID: \"3996e964-79db-4b61-b069-96e161ad78b9\") " Oct 30 18:29:13.220150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.219947 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.220847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.220188 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.220847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.220245 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.220847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.220377 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.220847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.220563 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:13.221300 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.221264 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.221557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.221307 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3996e964-79db-4b61-b069-96e161ad78b9-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.221557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.221323 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.221557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.221338 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.221557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.221353 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.225397 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.225281 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "kube-api-access-m7q4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.226496 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.226447 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.226876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.226852 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.227980 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.227958 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo" pod "3996e964-79db-4b61-b069-96e161ad78b9" (UID: "3996e964-79db-4b61-b069-96e161ad78b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:13.257023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.256995 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:29:13.322280 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322252 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322293 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322364 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322419 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322452 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322482 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322518 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322578 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322635 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\") pod \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\" (UID: \"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6\") " Oct 30 18:29:13.322666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.322633 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.323079 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.323054 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.324220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.323400 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:13.324220 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.323752 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.324802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.324758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.325169 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "kube-api-access-k8cg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.325189 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326502 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7q4k\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-kube-api-access-m7q4k\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326523 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326536 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326547 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3996e964-79db-4b61-b069-96e161ad78b9-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326560 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326572 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.326665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326584 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3996e964-79db-4b61-b069-96e161ad78b9-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.327103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326676 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.327103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326693 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8cg6\" (UniqueName: \"kubernetes.io/projected/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-kube-api-access-k8cg6\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.327103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326717 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.327103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326730 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3996e964-79db-4b61-b069-96e161ad78b9-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.327103 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.326794 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.327522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.327497 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo" pod "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" (UID: "f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:13.430752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.429703 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.430752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.429742 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.435332 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:29:13.435291 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598deb48_d868_41c4_a40f_b5909b99702d.slice/crio-0dc83833d834ca9c6d0e7fe69b40b6c0a604a833288035042d284f3ef040cc22 WatchSource:0}: Error finding container 0dc83833d834ca9c6d0e7fe69b40b6c0a604a833288035042d284f3ef040cc22: Status 404 returned error can't find the container with id 0dc83833d834ca9c6d0e7fe69b40b6c0a604a833288035042d284f3ef040cc22 Oct 30 18:29:13.437725 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.437691 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:29:13.522802 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.522768 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:29:13.631493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631403 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631493 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631486 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631682 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631770 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631741 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631797 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631829 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.631886 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631839 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.632051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631908 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.632051 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.631952 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.632149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632050 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.632149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632050 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\") pod \"9857f8a3-feef-4a67-8034-3a829c5239db\" (UID: \"9857f8a3-feef-4a67-8034-3a829c5239db\") " Oct 30 18:29:13.632149 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632128 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:13.632352 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632312 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.632559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632514 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.632822 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632802 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.632902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632823 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9857f8a3-feef-4a67-8034-3a829c5239db-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.632902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632838 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.632902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632852 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.632902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.632868 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.634064 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.634037 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.634178 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.634159 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.634834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.634806 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:13.634834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.634824 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp" pod "9857f8a3-feef-4a67-8034-3a829c5239db" (UID: "9857f8a3-feef-4a67-8034-3a829c5239db"). InnerVolumeSpecName "kube-api-access-wqgwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.733734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.733699 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9857f8a3-feef-4a67-8034-3a829c5239db-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.733734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.733734 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgwp\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-kube-api-access-wqgwp\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.733972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.733746 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9857f8a3-feef-4a67-8034-3a829c5239db-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.733972 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.733758 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9857f8a3-feef-4a67-8034-3a829c5239db-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.853704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.853660 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" event={"ID":"598deb48-d868-41c4-a40f-b5909b99702d","Type":"ContainerStarted","Data":"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4"} Oct 30 18:29:13.854191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.853722 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" event={"ID":"598deb48-d868-41c4-a40f-b5909b99702d","Type":"ContainerStarted","Data":"0dc83833d834ca9c6d0e7fe69b40b6c0a604a833288035042d284f3ef040cc22"} Oct 30 18:29:13.856081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.856047 5024 generic.go:356] "Generic (PLEG): container finished" podID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerID="da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6" exitCode=0 Oct 30 18:29:13.856259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.856115 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" Oct 30 18:29:13.856259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.856177 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" event={"ID":"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6","Type":"ContainerDied","Data":"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6"} Oct 30 18:29:13.856374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.856314 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp" event={"ID":"f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6","Type":"ContainerDied","Data":"d4d175423f8c5823eef5fb505d8d14d89e89108f0ec65706184114fe241a7be8"} Oct 30 18:29:13.856374 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.856347 5024 scope.go:122] "RemoveContainer" containerID="da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6" Oct 30 18:29:13.858507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.858484 5024 generic.go:356] "Generic (PLEG): container finished" podID="3996e964-79db-4b61-b069-96e161ad78b9" containerID="c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5" exitCode=0 Oct 30 18:29:13.858747 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.858724 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" Oct 30 18:29:13.859091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.858981 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" event={"ID":"3996e964-79db-4b61-b069-96e161ad78b9","Type":"ContainerDied","Data":"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5"} Oct 30 18:29:13.859091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.859036 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6" event={"ID":"3996e964-79db-4b61-b069-96e161ad78b9","Type":"ContainerDied","Data":"0b7dbad7e75f57bd188ee8809ba2b0ac4a82400def420eb6a50d328100e5a936"} Oct 30 18:29:13.860876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.860851 5024 generic.go:356] "Generic (PLEG): container finished" podID="9857f8a3-feef-4a67-8034-3a829c5239db" containerID="d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b" exitCode=0 Oct 30 18:29:13.860979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.860954 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" event={"ID":"9857f8a3-feef-4a67-8034-3a829c5239db","Type":"ContainerDied","Data":"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b"} Oct 30 18:29:13.861025 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.860994 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" event={"ID":"9857f8a3-feef-4a67-8034-3a829c5239db","Type":"ContainerDied","Data":"913397ed36d1e7fc1ef5ca00427307c4e44027c1a953f1db5fafe33bde0426aa"} Oct 30 18:29:13.861073 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.861055 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn" Oct 30 18:29:13.872772 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.872730 5024 scope.go:122] "RemoveContainer" containerID="da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6" Oct 30 18:29:13.873298 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:13.873265 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6\": container with ID starting with da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6 not found: ID does not exist" containerID="da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6" Oct 30 18:29:13.873387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.873312 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6"} err="failed to get container status \"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6\": rpc error: code = NotFound desc = could not find container \"da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6\": container with ID starting with da16adb43a50e76c599efbadad247e95bbe331ff736db468e7631e48d8e2fab6 not found: ID does not exist" Oct 30 18:29:13.873387 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.873338 5024 scope.go:122] "RemoveContainer" containerID="c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5" Oct 30 18:29:13.886498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.886461 5024 scope.go:122] "RemoveContainer" containerID="c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5" Oct 30 18:29:13.887088 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:13.887057 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5\": container with ID starting with c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5 not found: ID does not exist" containerID="c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5" Oct 30 18:29:13.887212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.887099 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5"} err="failed to get container status \"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5\": rpc error: code = NotFound desc = could not find container \"c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5\": container with ID starting with c32b3babc8e09a0e98eecffbdf62148d9f2b7fea74ced4053f2adefe25e23de5 not found: ID does not exist" Oct 30 18:29:13.887212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.887126 5024 scope.go:122] "RemoveContainer" containerID="d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b" Oct 30 18:29:13.896087 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.896038 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" podStartSLOduration=1.896020895 podStartE2EDuration="1.896020895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:13.891561471 +0000 UTC m=+1704.712273249" watchObservedRunningTime="2026-10-30 18:29:13.896020895 +0000 UTC m=+1704.716732669" Oct 30 18:29:13.898654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.898585 5024 scope.go:122] "RemoveContainer" containerID="d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b" Oct 30 18:29:13.899030 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:13.899008 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b\": container with ID starting with d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b not found: ID does not exist" containerID="d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b" Oct 30 18:29:13.899111 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.899038 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b"} err="failed to get container status \"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b\": rpc error: code = NotFound desc = could not find container \"d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b\": container with ID starting with d957162c4dbfb7d31907176727c3e7e177aad5049e11ec762a28c3f6baafb29b not found: ID does not exist" Oct 30 18:29:13.927125 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.927090 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:29:13.931665 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.931629 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-8htdn"] Oct 30 18:29:13.962226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.962187 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:29:13.966739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.966706 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-22bnp"] Oct 30 18:29:13.997366 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:13.997336 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:29:14.002339 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:14.002304 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-m54h6"] Oct 30 18:29:14.075748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:14.075705 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:15.306262 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:15.306223 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3996e964-79db-4b61-b069-96e161ad78b9" path="/var/lib/kubelet/pods/3996e964-79db-4b61-b069-96e161ad78b9/volumes" Oct 30 18:29:15.306749 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:15.306733 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" path="/var/lib/kubelet/pods/9857f8a3-feef-4a67-8034-3a829c5239db/volumes" Oct 30 18:29:15.307185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:15.307172 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" path="/var/lib/kubelet/pods/f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6/volumes" Oct 30 18:29:23.073719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:23.073677 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:23.087355 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:23.087322 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:23.949692 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:23.949658 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:29:24.721245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:24.721189 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:29:24.722337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:24.722265 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" containerID="cri-o://1ffb98bf9a0c686e9177d54a15ccc8a82aa7b95f42ded97436e60fc3834bafb3" gracePeriod=30 Oct 30 18:29:24.723644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:24.723238 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:29:24.723644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:24.723399 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" containerID="cri-o://839421118a5d4e29abe432b8ed169b46ed94654ede5bd81e144f82606a2944e7" gracePeriod=30 Oct 30 18:29:29.164778 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:29.164742 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:29:29.982364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:29.982264 5024 generic.go:356] "Generic (PLEG): container finished" podID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerID="1ffb98bf9a0c686e9177d54a15ccc8a82aa7b95f42ded97436e60fc3834bafb3" exitCode=0 Oct 30 18:29:29.982364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:29.982292 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" event={"ID":"7ba66fe8-ed81-4310-b406-9a643f2b2ba6","Type":"ContainerDied","Data":"1ffb98bf9a0c686e9177d54a15ccc8a82aa7b95f42ded97436e60fc3834bafb3"} Oct 30 18:29:29.984294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:29.984263 5024 generic.go:356] "Generic (PLEG): container finished" podID="88df402f-fe7b-422f-89b6-96933e761516" containerID="839421118a5d4e29abe432b8ed169b46ed94654ede5bd81e144f82606a2944e7" exitCode=0 Oct 30 18:29:29.984426 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:29.984317 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" event={"ID":"88df402f-fe7b-422f-89b6-96933e761516","Type":"ContainerDied","Data":"839421118a5d4e29abe432b8ed169b46ed94654ede5bd81e144f82606a2944e7"} Oct 30 18:29:30.042305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.042277 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:29:30.088529 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.088494 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:29:30.224687 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224647 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224714 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224741 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224803 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224826 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224864 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224889 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224922 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224932 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.224964 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225044 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225031 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225121 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225157 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225197 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225222 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225249 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225291 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\") pod \"88df402f-fe7b-422f-89b6-96933e761516\" (UID: \"88df402f-fe7b-422f-89b6-96933e761516\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225385 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225458 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\") pod \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\" (UID: \"7ba66fe8-ed81-4310-b406-9a643f2b2ba6\") " Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225452 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.225733 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.226029 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.226276 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.226431 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.227211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.226577 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.227927 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228072 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228238 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228431 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228531 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "kube-api-access-wkskv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228547 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228564 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228574 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "kube-api-access-9hpx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.228782 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228777 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228799 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228813 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88df402f-fe7b-422f-89b6-96933e761516-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228826 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88df402f-fe7b-422f-89b6-96933e761516-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228837 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228846 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228854 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228862 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228870 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228878 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228887 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.229458 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.228895 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.230202 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.229945 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy" pod "88df402f-fe7b-422f-89b6-96933e761516" (UID: "88df402f-fe7b-422f-89b6-96933e761516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.230767 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.230745 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo" pod "7ba66fe8-ed81-4310-b406-9a643f2b2ba6" (UID: "7ba66fe8-ed81-4310-b406-9a643f2b2ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330072 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330102 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hpx5\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-kube-api-access-9hpx5\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330113 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330124 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkskv\" (UniqueName: \"kubernetes.io/projected/88df402f-fe7b-422f-89b6-96933e761516-kube-api-access-wkskv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330135 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88df402f-fe7b-422f-89b6-96933e761516-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.330169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.330144 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ba66fe8-ed81-4310-b406-9a643f2b2ba6-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.994176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.994139 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" Oct 30 18:29:30.994176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.994168 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4" event={"ID":"88df402f-fe7b-422f-89b6-96933e761516","Type":"ContainerDied","Data":"4eb7579379ec794c49b335418d9006796aa1512ddd508454ed7efdd89b74442d"} Oct 30 18:29:30.994486 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:30.994238 5024 scope.go:122] "RemoveContainer" containerID="839421118a5d4e29abe432b8ed169b46ed94654ede5bd81e144f82606a2944e7" Oct 30 18:29:31.006124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.001059 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" event={"ID":"7ba66fe8-ed81-4310-b406-9a643f2b2ba6","Type":"ContainerDied","Data":"e6874e633c0e42a063c8e1ced1d9cc5900255413604d0554785aa157a72cc281"} Oct 30 18:29:31.006124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.001178 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh" Oct 30 18:29:31.013398 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.013376 5024 scope.go:122] "RemoveContainer" containerID="1ffb98bf9a0c686e9177d54a15ccc8a82aa7b95f42ded97436e60fc3834bafb3" Oct 30 18:29:31.057898 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.057835 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:29:31.068177 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.068133 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-npxx4"] Oct 30 18:29:31.101628 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.101571 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:29:31.107927 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.107894 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-g2tbh"] Oct 30 18:29:31.306099 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.306020 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" path="/var/lib/kubelet/pods/7ba66fe8-ed81-4310-b406-9a643f2b2ba6/volumes" Oct 30 18:29:31.306496 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:31.306482 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88df402f-fe7b-422f-89b6-96933e761516" path="/var/lib/kubelet/pods/88df402f-fe7b-422f-89b6-96933e761516/volumes" Oct 30 18:29:40.456838 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:40.456782 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:29:40.787519 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:40.787425 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:29:44.188067 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.188008 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" containerID="cri-o://5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe" gracePeriod=45 Oct 30 18:29:44.466947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.466916 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:29:44.606145 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606105 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606285 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\" (UniqueName: \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606330 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\" (UniqueName: \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606371 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606447 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\") pod \"38680810-b7c9-485b-87eb-b5c34b9d1436\" (UID: \"38680810-b7c9-485b-87eb-b5c34b9d1436\") " Oct 30 18:29:44.606530 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606506 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.606839 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606815 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.606907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606815 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.606961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606953 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.607010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606970 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.607010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.606982 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.608946 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.608915 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "kube-api-access-hf86t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:44.609135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.609115 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:44.609212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.609128 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm" pod "38680810-b7c9-485b-87eb-b5c34b9d1436" (UID: "38680810-b7c9-485b-87eb-b5c34b9d1436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.708210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.708127 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38680810-b7c9-485b-87eb-b5c34b9d1436-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.708210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.708161 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf86t\" (UniqueName: \"kubernetes.io/projected/38680810-b7c9-485b-87eb-b5c34b9d1436-kube-api-access-hf86t\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.708210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:44.708173 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38680810-b7c9-485b-87eb-b5c34b9d1436-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.087240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.087139 5024 generic.go:356] "Generic (PLEG): container finished" podID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerID="5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe" exitCode=0 Oct 30 18:29:45.087240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.087194 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" event={"ID":"38680810-b7c9-485b-87eb-b5c34b9d1436","Type":"ContainerDied","Data":"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe"} Oct 30 18:29:45.087240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.087225 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" event={"ID":"38680810-b7c9-485b-87eb-b5c34b9d1436","Type":"ContainerDied","Data":"645cdcee0233842d6c1dbeab4254ad13234985fadbfcbe146f90ea1fe3d18f9a"} Oct 30 18:29:45.087240 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.087242 5024 scope.go:122] "RemoveContainer" containerID="5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe" Oct 30 18:29:45.087551 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.087197 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq" Oct 30 18:29:45.097712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.097685 5024 scope.go:122] "RemoveContainer" containerID="5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe" Oct 30 18:29:45.098063 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:45.098044 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe\": container with ID starting with 5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe not found: ID does not exist" containerID="5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe" Oct 30 18:29:45.098117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.098081 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe"} err="failed to get container status \"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe\": rpc error: code = NotFound desc = could not find container \"5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe\": container with ID starting with 5f55cfefa529da15563ce722369f614402056b9054f92d8fc958e65aac1a42fe not found: ID does not exist" Oct 30 18:29:45.138965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.138926 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:29:45.148278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.148241 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/premium-w1-fb2fa0-kserve-69ddbd9d58-hl2mq"] Oct 30 18:29:45.305512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:45.305475 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" path="/var/lib/kubelet/pods/38680810-b7c9-485b-87eb-b5c34b9d1436/volumes" Oct 30 18:29:55.811672 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:55.811614 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" containerID="cri-o://ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79" gracePeriod=45 Oct 30 18:29:56.083250 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.083226 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:29:56.164120 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.164078 5024 generic.go:356] "Generic (PLEG): container finished" podID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerID="ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79" exitCode=0 Oct 30 18:29:56.164324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.164135 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" Oct 30 18:29:56.164324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.164160 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" event={"ID":"8396c956-04cc-4dc0-8abd-8103d8ef1ca4","Type":"ContainerDied","Data":"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79"} Oct 30 18:29:56.164324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.164203 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2" event={"ID":"8396c956-04cc-4dc0-8abd-8103d8ef1ca4","Type":"ContainerDied","Data":"91f9c0c4daab7ccfa77564125d9d32c42565dfe6f886c91628dc9248b5e4c37e"} Oct 30 18:29:56.164324 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.164225 5024 scope.go:122] "RemoveContainer" containerID="ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79" Oct 30 18:29:56.176674 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.176575 5024 scope.go:122] "RemoveContainer" containerID="ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79" Oct 30 18:29:56.177122 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:29:56.177083 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79\": container with ID starting with ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79 not found: ID does not exist" containerID="ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79" Oct 30 18:29:56.177234 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.177134 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79"} err="failed to get container status \"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79\": rpc error: code = NotFound desc = could not find container \"ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79\": container with ID starting with ec6bb8beaea8885bbf22a2814d04b0fa6cccb0e86a58ecd78c8b3d219209fc79 not found: ID does not exist" Oct 30 18:29:56.256787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.256755 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.256991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.256831 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.256991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.256861 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.257115 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.256990 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.257115 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257022 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\" (UniqueName: \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.257227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257116 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\") pod \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\" (UID: \"8396c956-04cc-4dc0-8abd-8103d8ef1ca4\") " Oct 30 18:29:56.257227 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257127 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.257341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257217 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.257341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257323 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.257683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257570 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.257683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257666 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.257683 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.257681 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.261778 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.261748 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:56.261953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.261781 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "kube-api-access-j6t78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:56.261953 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.261858 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm" pod "8396c956-04cc-4dc0-8abd-8103d8ef1ca4" (UID: "8396c956-04cc-4dc0-8abd-8103d8ef1ca4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.358337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.358254 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6t78\" (UniqueName: \"kubernetes.io/projected/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-kube-api-access-j6t78\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.358337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.358282 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.358337 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.358293 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8396c956-04cc-4dc0-8abd-8103d8ef1ca4-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.508091 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.508054 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:29:56.511847 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:56.511816 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/simulator-w3-8b6cf1-kserve-678594f54b-m5nh2"] Oct 30 18:29:57.305724 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:29:57.305687 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" path="/var/lib/kubelet/pods/8396c956-04cc-4dc0-8abd-8103d8ef1ca4/volumes" Oct 30 18:30:00.195609 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.194234 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:00.196903 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196877 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196915 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196945 5024 cpu_manager.go:414] "containerMap: removing container" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196952 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196986 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.196994 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197009 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197016 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197028 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197035 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197051 5024 cpu_manager.go:414] "containerMap: removing container" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197058 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197069 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197076 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197234 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3996e964-79db-4b61-b069-96e161ad78b9" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197249 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81a4e6c-0678-4ac5-acd2-8c0bda7fc8d6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197259 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="38680810-b7c9-485b-87eb-b5c34b9d1436" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197266 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba66fe8-ed81-4310-b406-9a643f2b2ba6" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197278 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8721759e-89db-4bf6-9ec1-58874188efb3" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197289 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8396c956-04cc-4dc0-8abd-8103d8ef1ca4" containerName="main" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197300 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="88df402f-fe7b-422f-89b6-96933e761516" containerName="istio-proxy" Oct 30 18:30:00.197382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.197311 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9857f8a3-feef-4a67-8034-3a829c5239db" containerName="istio-proxy" Oct 30 18:30:00.204907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.204871 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.205744 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.205716 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:00.211888 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.211859 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.217225 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.217198 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:00.235321 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.235285 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:00.235821 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.235796 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:30:00.317678 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.317641 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.317860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.317797 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssjp7\" (UniqueName: \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.317860 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.317845 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhnn\" (UniqueName: \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.317963 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.317947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.329429 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.324697 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j"] Oct 30 18:30:00.344977 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.344931 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j"] Oct 30 18:30:00.345244 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.345222 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.362866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.362833 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:00.418666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418629 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.418858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418692 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.418858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418736 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.418858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418786 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqz7m\" (UniqueName: \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.418858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418810 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.418858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.418845 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssjp7\" (UniqueName: \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.419127 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.419101 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhnn\" (UniqueName: \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.419318 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.419298 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.419474 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.419458 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.447294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.447211 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhnn\" (UniqueName: \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.447294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.447229 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssjp7\" (UniqueName: \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.520159 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.520124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.520369 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.520253 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqz7m\" (UniqueName: \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.520369 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.520286 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.520947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.520925 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.522975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.522955 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.530964 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.530940 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:00.539420 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.539398 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqz7m\" (UniqueName: \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\") pod \"maas-api-revoke-sub-63d25855-l484j\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.576213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.576181 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:00.666005 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.665978 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:00.850153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.849913 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:00.855535 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:30:00.853432 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed85308_c1aa_414c_a917_c1b86312926e.slice/crio-2e6d091212eaedff00e00d53d52cae00cddfef9c4e1968db2a96802831c76ba1 WatchSource:0}: Error finding container 2e6d091212eaedff00e00d53d52cae00cddfef9c4e1968db2a96802831c76ba1: Status 404 returned error can't find the container with id 2e6d091212eaedff00e00d53d52cae00cddfef9c4e1968db2a96802831c76ba1 Oct 30 18:30:00.856866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.855871 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:30:00.902706 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:00.902667 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:00.903863 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:30:00.903840 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85b4cad_d13c_4257_b1f8_db84d996bea5.slice/crio-b4d628601e2b62e79f235b95abea821089b55861c3a0b802bbf1c551916f2a06 WatchSource:0}: Error finding container b4d628601e2b62e79f235b95abea821089b55861c3a0b802bbf1c551916f2a06: Status 404 returned error can't find the container with id b4d628601e2b62e79f235b95abea821089b55861c3a0b802bbf1c551916f2a06 Oct 30 18:30:01.006135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:01.005861 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j"] Oct 30 18:30:01.009518 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:30:01.009475 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83302028_01b9_46d5_83d1_a2f41dae05ec.slice/crio-914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f WatchSource:0}: Error finding container 914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f: Status 404 returned error can't find the container with id 914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f Oct 30 18:30:01.203734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:01.203692 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" event={"ID":"83302028-01b9-46d5-83d1-a2f41dae05ec","Type":"ContainerStarted","Data":"914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f"} Oct 30 18:30:01.205090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:01.205063 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" event={"ID":"1ed85308-c1aa-414c-a917-c1b86312926e","Type":"ContainerStarted","Data":"2e6d091212eaedff00e00d53d52cae00cddfef9c4e1968db2a96802831c76ba1"} Oct 30 18:30:01.206191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:01.206170 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" event={"ID":"f85b4cad-d13c-4257-b1f8-db84d996bea5","Type":"ContainerStarted","Data":"b4d628601e2b62e79f235b95abea821089b55861c3a0b802bbf1c551916f2a06"} Oct 30 18:30:02.022710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.022622 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:30:02.030395 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.025224 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" podUID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" containerName="maas-api" containerID="cri-o://0a58a739d61f38317dfc16cc900ee14dae1c2c08fcf8736dd6201d59f2a7e23e" gracePeriod=30 Oct 30 18:30:02.236654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.234686 5024 generic.go:356] "Generic (PLEG): container finished" podID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" containerID="0a58a739d61f38317dfc16cc900ee14dae1c2c08fcf8736dd6201d59f2a7e23e" exitCode=0 Oct 30 18:30:02.236654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.234775 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" event={"ID":"4583e34d-8f75-4ab2-9ae4-f0a542175d1c","Type":"ContainerDied","Data":"0a58a739d61f38317dfc16cc900ee14dae1c2c08fcf8736dd6201d59f2a7e23e"} Oct 30 18:30:02.414789 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.414749 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:30:02.467836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.466737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\") pod \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " Oct 30 18:30:02.467836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.466843 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\" (UniqueName: \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\") pod \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " Oct 30 18:30:02.467836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.466941 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\") pod \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " Oct 30 18:30:02.467836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.467014 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\") pod \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\" (UID: \"4583e34d-8f75-4ab2-9ae4-f0a542175d1c\") " Oct 30 18:30:02.474798 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.474749 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca" pod "4583e34d-8f75-4ab2-9ae4-f0a542175d1c" (UID: "4583e34d-8f75-4ab2-9ae4-f0a542175d1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:02.475936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.475908 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z" pod "4583e34d-8f75-4ab2-9ae4-f0a542175d1c" (UID: "4583e34d-8f75-4ab2-9ae4-f0a542175d1c"). InnerVolumeSpecName "kube-api-access-8688z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:02.480175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.480149 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs" pod "4583e34d-8f75-4ab2-9ae4-f0a542175d1c" (UID: "4583e34d-8f75-4ab2-9ae4-f0a542175d1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:02.480875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.480820 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls" pod "4583e34d-8f75-4ab2-9ae4-f0a542175d1c" (UID: "4583e34d-8f75-4ab2-9ae4-f0a542175d1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:02.568652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.568553 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8688z\" (UniqueName: \"kubernetes.io/projected/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-kube-api-access-8688z\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:02.568652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.568588 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:02.568652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.568630 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:02.568652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:02.568638 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4583e34d-8f75-4ab2-9ae4-f0a542175d1c-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:03.261412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:03.261378 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" event={"ID":"4583e34d-8f75-4ab2-9ae4-f0a542175d1c","Type":"ContainerDied","Data":"d34cf1fafc58f48799df355defa5954fbb7195d9b2e552beb0adf8a294cd7102"} Oct 30 18:30:03.261852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:03.261450 5024 scope.go:122] "RemoveContainer" containerID="0a58a739d61f38317dfc16cc900ee14dae1c2c08fcf8736dd6201d59f2a7e23e" Oct 30 18:30:03.261852 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:03.261575 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6" Oct 30 18:30:03.355746 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:03.355705 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:30:03.418006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:03.417956 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2fa0-6696677465-t6sq6"] Oct 30 18:30:05.306316 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:05.306275 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" path="/var/lib/kubelet/pods/4583e34d-8f75-4ab2-9ae4-f0a542175d1c/volumes" Oct 30 18:30:05.312851 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:05.312819 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" event={"ID":"1ed85308-c1aa-414c-a917-c1b86312926e","Type":"ContainerStarted","Data":"4186db4538dd15786a5ba81c5c1f9b8ea4b938eeeacb7aff8712d7c3d9edaafc"} Oct 30 18:30:05.350417 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:05.350345 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" podStartSLOduration=1.393120185 podStartE2EDuration="5.35032925s" totalImagesPullingTime="3.957209065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:05.349030526 +0000 UTC m=+1756.169742307" watchObservedRunningTime="2026-10-30 18:30:05.35032925 +0000 UTC m=+1756.171041044" Oct 30 18:30:06.322671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.322198 5024 generic.go:356] "Generic (PLEG): container finished" podID="1ed85308-c1aa-414c-a917-c1b86312926e" containerID="4186db4538dd15786a5ba81c5c1f9b8ea4b938eeeacb7aff8712d7c3d9edaafc" exitCode=60 Oct 30 18:30:06.322671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.322456 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" event={"ID":"1ed85308-c1aa-414c-a917-c1b86312926e","Type":"ContainerDied","Data":"4186db4538dd15786a5ba81c5c1f9b8ea4b938eeeacb7aff8712d7c3d9edaafc"} Oct 30 18:30:06.322671 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.322567 5024 scope.go:122] "RemoveContainer" containerID="4186db4538dd15786a5ba81c5c1f9b8ea4b938eeeacb7aff8712d7c3d9edaafc" Oct 30 18:30:06.326333 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.326302 5024 generic.go:356] "Generic (PLEG): container finished" podID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerID="40567aff3bd34d1c198dfb895c39d5045624a1310dbedff60bb6350220dbdc74" exitCode=60 Oct 30 18:30:06.326457 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.326415 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" event={"ID":"f85b4cad-d13c-4257-b1f8-db84d996bea5","Type":"ContainerDied","Data":"40567aff3bd34d1c198dfb895c39d5045624a1310dbedff60bb6350220dbdc74"} Oct 30 18:30:06.326730 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.326712 5024 scope.go:122] "RemoveContainer" containerID="40567aff3bd34d1c198dfb895c39d5045624a1310dbedff60bb6350220dbdc74" Oct 30 18:30:06.329827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.329804 5024 generic.go:356] "Generic (PLEG): container finished" podID="83302028-01b9-46d5-83d1-a2f41dae05ec" containerID="6f4a6b69369de1e7fe8b0673f102f12564629d9bfe3091316558b2f033c0f6ae" exitCode=0 Oct 30 18:30:06.329933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:06.329880 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" event={"ID":"83302028-01b9-46d5-83d1-a2f41dae05ec","Type":"ContainerDied","Data":"6f4a6b69369de1e7fe8b0673f102f12564629d9bfe3091316558b2f033c0f6ae"} Oct 30 18:30:07.337394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.337362 5024 generic.go:356] "Generic (PLEG): container finished" podID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerID="959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04" exitCode=60 Oct 30 18:30:07.337873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.337420 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" event={"ID":"f85b4cad-d13c-4257-b1f8-db84d996bea5","Type":"ContainerDied","Data":"959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04"} Oct 30 18:30:07.337873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.337477 5024 scope.go:122] "RemoveContainer" containerID="40567aff3bd34d1c198dfb895c39d5045624a1310dbedff60bb6350220dbdc74" Oct 30 18:30:07.337873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.337726 5024 scope.go:122] "RemoveContainer" containerID="959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04" Oct 30 18:30:07.338036 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:07.337992 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq_odh-ai-gateway-infra(f85b4cad-d13c-4257-b1f8-db84d996bea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" Oct 30 18:30:07.339707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.339686 5024 generic.go:356] "Generic (PLEG): container finished" podID="1ed85308-c1aa-414c-a917-c1b86312926e" containerID="3aa58ff6c9ee29939c1e65a172e828950cf006b2ebbac04f6d1b3329317a033f" exitCode=60 Oct 30 18:30:07.339816 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.339715 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" event={"ID":"1ed85308-c1aa-414c-a917-c1b86312926e","Type":"ContainerDied","Data":"3aa58ff6c9ee29939c1e65a172e828950cf006b2ebbac04f6d1b3329317a033f"} Oct 30 18:30:07.339969 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.339953 5024 scope.go:122] "RemoveContainer" containerID="3aa58ff6c9ee29939c1e65a172e828950cf006b2ebbac04f6d1b3329317a033f" Oct 30 18:30:07.340205 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:07.340186 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d_odh-ai-gateway-infra(1ed85308-c1aa-414c-a917-c1b86312926e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" Oct 30 18:30:07.352112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.352089 5024 scope.go:122] "RemoveContainer" containerID="4186db4538dd15786a5ba81c5c1f9b8ea4b938eeeacb7aff8712d7c3d9edaafc" Oct 30 18:30:07.509218 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.509193 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:07.639090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.639051 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\") pod \"83302028-01b9-46d5-83d1-a2f41dae05ec\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " Oct 30 18:30:07.639296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.639158 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\" (UniqueName: \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\") pod \"83302028-01b9-46d5-83d1-a2f41dae05ec\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " Oct 30 18:30:07.639296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.639189 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\") pod \"83302028-01b9-46d5-83d1-a2f41dae05ec\" (UID: \"83302028-01b9-46d5-83d1-a2f41dae05ec\") " Oct 30 18:30:07.639679 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.639653 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca" pod "83302028-01b9-46d5-83d1-a2f41dae05ec" (UID: "83302028-01b9-46d5-83d1-a2f41dae05ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:07.641557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.641525 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m" pod "83302028-01b9-46d5-83d1-a2f41dae05ec" (UID: "83302028-01b9-46d5-83d1-a2f41dae05ec"). InnerVolumeSpecName "kube-api-access-qqz7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:07.641884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.641865 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca" pod "83302028-01b9-46d5-83d1-a2f41dae05ec" (UID: "83302028-01b9-46d5-83d1-a2f41dae05ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:07.741036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.740991 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqz7m\" (UniqueName: \"kubernetes.io/projected/83302028-01b9-46d5-83d1-a2f41dae05ec-kube-api-access-qqz7m\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:07.741036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.741030 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:07.741036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:07.741040 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83302028-01b9-46d5-83d1-a2f41dae05ec-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:08.348408 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:08.348378 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" Oct 30 18:30:08.348967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:08.348411 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j" event={"ID":"83302028-01b9-46d5-83d1-a2f41dae05ec","Type":"ContainerDied","Data":"914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f"} Oct 30 18:30:08.348967 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:08.348460 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914c5b237ad46217ac71175793558cf98ec3fe498b3c42c1a4c9c40f50e2835f" Oct 30 18:30:08.350415 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:08.350395 5024 scope.go:122] "RemoveContainer" containerID="3aa58ff6c9ee29939c1e65a172e828950cf006b2ebbac04f6d1b3329317a033f" Oct 30 18:30:08.350766 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:08.350723 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d_odh-ai-gateway-infra(1ed85308-c1aa-414c-a917-c1b86312926e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" Oct 30 18:30:08.352188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:08.352169 5024 scope.go:122] "RemoveContainer" containerID="959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04" Oct 30 18:30:08.352366 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:08.352349 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq_odh-ai-gateway-infra(f85b4cad-d13c-4257-b1f8-db84d996bea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" Oct 30 18:30:10.461178 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:10.461138 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:30:10.754221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:10.754120 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:30:10.754391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:10.754357 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" containerID="cri-o://06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26" gracePeriod=30 Oct 30 18:30:15.826059 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:15.826025 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:30:15.826461 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:15.826247 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" podUID="86ee8676-bb5c-455e-9456-4b1d54381b92" containerName="maas-api" containerID="cri-o://2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf" gracePeriod=30 Oct 30 18:30:15.894188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:15.894155 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:16.090960 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.089936 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:16.149742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.149035 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\" (UniqueName: \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\") pod \"1ed85308-c1aa-414c-a917-c1b86312926e\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " Oct 30 18:30:16.149742 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.149118 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\") pod \"1ed85308-c1aa-414c-a917-c1b86312926e\" (UID: \"1ed85308-c1aa-414c-a917-c1b86312926e\") " Oct 30 18:30:16.149986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.149829 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca" pod "1ed85308-c1aa-414c-a917-c1b86312926e" (UID: "1ed85308-c1aa-414c-a917-c1b86312926e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.171148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.171097 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7" pod "1ed85308-c1aa-414c-a917-c1b86312926e" (UID: "1ed85308-c1aa-414c-a917-c1b86312926e"). InnerVolumeSpecName "kube-api-access-ssjp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.244586 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.244269 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:30:16.250991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.250930 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssjp7\" (UniqueName: \"kubernetes.io/projected/1ed85308-c1aa-414c-a917-c1b86312926e-kube-api-access-ssjp7\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.250991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.250966 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ed85308-c1aa-414c-a917-c1b86312926e-maas-api-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.283441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.283410 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:30:16.351695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351591 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.351695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351662 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351703 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\") pod \"86ee8676-bb5c-455e-9456-4b1d54381b92\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351734 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351791 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") pod \"86ee8676-bb5c-455e-9456-4b1d54381b92\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351855 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351884 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351907 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.351945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.351921 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352021 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352075 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\") pod \"86ee8676-bb5c-455e-9456-4b1d54381b92\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352109 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352117 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352155 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\") pod \"86ee8676-bb5c-455e-9456-4b1d54381b92\" (UID: \"86ee8676-bb5c-455e-9456-4b1d54381b92\") " Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.352306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352285 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\") pod \"7de5521e-2902-4f61-9eda-b7d585099b37\" (UID: \"7de5521e-2902-4f61-9eda-b7d585099b37\") " Oct 30 18:30:16.352710 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.352613 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.353142 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.353114 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.353264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.353145 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de5521e-2902-4f61-9eda-b7d585099b37-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.353264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.353158 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.353264 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.353171 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.353681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.353656 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.354654 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.354586 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88" pod "86ee8676-bb5c-455e-9456-4b1d54381b92" (UID: "86ee8676-bb5c-455e-9456-4b1d54381b92"). InnerVolumeSpecName "kube-api-access-lsq88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.354793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.354772 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca" pod "86ee8676-bb5c-455e-9456-4b1d54381b92" (UID: "86ee8676-bb5c-455e-9456-4b1d54381b92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.355491 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.355471 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:16.356983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.356961 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "kube-api-access-wxp7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.357104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.356981 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs" pod "86ee8676-bb5c-455e-9456-4b1d54381b92" (UID: "86ee8676-bb5c-455e-9456-4b1d54381b92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:16.357104 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.357060 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls" pod "86ee8676-bb5c-455e-9456-4b1d54381b92" (UID: "86ee8676-bb5c-455e-9456-4b1d54381b92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:16.357226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.357018 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.358574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.358549 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token" pod "7de5521e-2902-4f61-9eda-b7d585099b37" (UID: "7de5521e-2902-4f61-9eda-b7d585099b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.405842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.403431 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" Oct 30 18:30:16.405842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.403463 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" event={"ID":"7de5521e-2902-4f61-9eda-b7d585099b37","Type":"ContainerDied","Data":"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26"} Oct 30 18:30:16.405842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.403522 5024 scope.go:122] "RemoveContainer" containerID="06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26" Oct 30 18:30:16.405842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.403433 5024 generic.go:356] "Generic (PLEG): container finished" podID="7de5521e-2902-4f61-9eda-b7d585099b37" containerID="06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26" exitCode=0 Oct 30 18:30:16.405842 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.403729 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k" event={"ID":"7de5521e-2902-4f61-9eda-b7d585099b37","Type":"ContainerDied","Data":"7ec50beb56442687283cd81f82535bf8e1145764e52775cced563061e95fd954"} Oct 30 18:30:16.411879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.411849 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" Oct 30 18:30:16.412388 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.412355 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d" event={"ID":"1ed85308-c1aa-414c-a917-c1b86312926e","Type":"ContainerDied","Data":"2e6d091212eaedff00e00d53d52cae00cddfef9c4e1968db2a96802831c76ba1"} Oct 30 18:30:16.413808 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.413784 5024 generic.go:356] "Generic (PLEG): container finished" podID="86ee8676-bb5c-455e-9456-4b1d54381b92" containerID="2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf" exitCode=0 Oct 30 18:30:16.413938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.413912 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" event={"ID":"86ee8676-bb5c-455e-9456-4b1d54381b92","Type":"ContainerDied","Data":"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf"} Oct 30 18:30:16.414010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.413939 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" event={"ID":"86ee8676-bb5c-455e-9456-4b1d54381b92","Type":"ContainerDied","Data":"870332b3881447ee17ce64c3cea321dc07dffb7afffa5d0e09a53028d6e9bc19"} Oct 30 18:30:16.414010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.413987 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g" Oct 30 18:30:16.420469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.420446 5024 scope.go:122] "RemoveContainer" containerID="06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26" Oct 30 18:30:16.420810 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:16.420771 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26\": container with ID starting with 06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26 not found: ID does not exist" containerID="06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26" Oct 30 18:30:16.420924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.420898 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26"} err="failed to get container status \"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26\": rpc error: code = NotFound desc = could not find container \"06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26\": container with ID starting with 06cfba7b4624cdaeff4227b3de1be543b116218242dbfa5430a149b3e5270b26 not found: ID does not exist" Oct 30 18:30:16.420991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.420928 5024 scope.go:122] "RemoveContainer" containerID="3aa58ff6c9ee29939c1e65a172e828950cf006b2ebbac04f6d1b3329317a033f" Oct 30 18:30:16.430072 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.430054 5024 scope.go:122] "RemoveContainer" containerID="2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf" Oct 30 18:30:16.441473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.441447 5024 scope.go:122] "RemoveContainer" containerID="2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf" Oct 30 18:30:16.441937 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:16.441905 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf\": container with ID starting with 2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf not found: ID does not exist" containerID="2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf" Oct 30 18:30:16.442039 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.441944 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf"} err="failed to get container status \"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf\": rpc error: code = NotFound desc = could not find container \"2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf\": container with ID starting with 2fa91bb1c95dc15b997e72693bd9e0d0b17514a38274ada00500f73e52398bdf not found: ID does not exist" Oct 30 18:30:16.454681 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454656 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454683 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsq88\" (UniqueName: \"kubernetes.io/projected/86ee8676-bb5c-455e-9456-4b1d54381b92-kube-api-access-lsq88\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454698 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de5521e-2902-4f61-9eda-b7d585099b37-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454712 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454723 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de5521e-2902-4f61-9eda-b7d585099b37-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454735 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxp7d\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-kube-api-access-wxp7d\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454748 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ee8676-bb5c-455e-9456-4b1d54381b92-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454767 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ee8676-bb5c-455e-9456-4b1d54381b92-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.454793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.454781 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de5521e-2902-4f61-9eda-b7d585099b37-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.482612 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.482542 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:30:16.486236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.486216 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-x768k"] Oct 30 18:30:16.515711 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.515679 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:16.521695 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.521669 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889750-rfh4d"] Oct 30 18:30:16.559188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.559144 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:30:16.563587 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:16.563559 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8b6cf1-5b6cbf66bb-x2z6g"] Oct 30 18:30:17.305614 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:17.305551 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" path="/var/lib/kubelet/pods/1ed85308-c1aa-414c-a917-c1b86312926e/volumes" Oct 30 18:30:17.306236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:17.306212 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" path="/var/lib/kubelet/pods/7de5521e-2902-4f61-9eda-b7d585099b37/volumes" Oct 30 18:30:17.306691 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:17.306670 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ee8676-bb5c-455e-9456-4b1d54381b92" path="/var/lib/kubelet/pods/86ee8676-bb5c-455e-9456-4b1d54381b92/volumes" Oct 30 18:30:21.858489 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.858448 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:30:21.859230 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859211 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859234 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859251 5024 cpu_manager.go:414] "containerMap: removing container" podUID="83302028-01b9-46d5-83d1-a2f41dae05ec" containerName="revoke-keys" Oct 30 18:30:21.859274 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859260 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83302028-01b9-46d5-83d1-a2f41dae05ec" containerName="revoke-keys" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859278 5024 cpu_manager.go:414] "containerMap: removing container" podUID="86ee8676-bb5c-455e-9456-4b1d54381b92" containerName="maas-api" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859287 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ee8676-bb5c-455e-9456-4b1d54381b92" containerName="maas-api" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859303 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859310 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859322 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859329 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859342 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" containerName="maas-api" Oct 30 18:30:21.859380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859349 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" containerName="maas-api" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859472 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859483 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="83302028-01b9-46d5-83d1-a2f41dae05ec" containerName="revoke-keys" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859495 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ee8676-bb5c-455e-9456-4b1d54381b92" containerName="maas-api" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859504 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de5521e-2902-4f61-9eda-b7d585099b37" containerName="istio-proxy" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859514 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed85308-c1aa-414c-a917-c1b86312926e" containerName="cleanup" Oct 30 18:30:21.859627 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.859523 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4583e34d-8f75-4ab2-9ae4-f0a542175d1c" containerName="maas-api" Oct 30 18:30:21.864988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.864964 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:21.882739 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.882701 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:30:21.888347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.888316 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c9a107\"/\"openshift-service-ca.crt\"" Oct 30 18:30:21.888347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.888342 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c9a107\"/\"kube-root-ca.crt\"" Oct 30 18:30:21.960757 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.960729 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c9a107\"/\"default-dockercfg-wn7mc\"" Oct 30 18:30:21.960974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:21.960778 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c9a107\"/\"test-model-c9a107-kserve-self-signed-certs\"" Oct 30 18:30:22.019712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019655 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.019712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019703 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfv66\" (UniqueName: \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.019996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019742 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.019996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019823 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.019996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019889 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.019996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.019907 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121509 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121560 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121769 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121802 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfv66\" (UniqueName: \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121847 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.121885 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.122357 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122562 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.122384 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.122833 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.122753 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.135854 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.135817 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.138865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.138829 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.145362 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.145328 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfv66\" (UniqueName: \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\") pod \"test-model-c9a107-kserve-56bb465b97-gnk49\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.180110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.180069 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:22.298406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.298375 5024 scope.go:122] "RemoveContainer" containerID="959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04" Oct 30 18:30:22.490285 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:22.490249 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:30:22.548032 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:30:22.548001 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb63940fb_7c10_4316_b7f8_46532f3d849d.slice/crio-459b40a4c3f4f65a8021a0f43bd5ea92885ed6c662914361157db2650627e61a WatchSource:0}: Error finding container 459b40a4c3f4f65a8021a0f43bd5ea92885ed6c662914361157db2650627e61a: Status 404 returned error can't find the container with id 459b40a4c3f4f65a8021a0f43bd5ea92885ed6c662914361157db2650627e61a Oct 30 18:30:23.459128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.459016 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" event={"ID":"b63940fb-7c10-4316-b7f8-46532f3d849d","Type":"ContainerStarted","Data":"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354"} Oct 30 18:30:23.459128 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.459070 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" event={"ID":"b63940fb-7c10-4316-b7f8-46532f3d849d","Type":"ContainerStarted","Data":"459b40a4c3f4f65a8021a0f43bd5ea92885ed6c662914361157db2650627e61a"} Oct 30 18:30:23.460965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.460938 5024 generic.go:356] "Generic (PLEG): container finished" podID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerID="86d0715f0b745015d42c6ad5112cf8c75f01b955b17113ba9533dcc78871f6ad" exitCode=60 Oct 30 18:30:23.461075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.461006 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" event={"ID":"f85b4cad-d13c-4257-b1f8-db84d996bea5","Type":"ContainerDied","Data":"86d0715f0b745015d42c6ad5112cf8c75f01b955b17113ba9533dcc78871f6ad"} Oct 30 18:30:23.461075 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.461061 5024 scope.go:122] "RemoveContainer" containerID="959b46d8a82bd6d130d4f59bd2a3fe944ec4be5e1961f335f2d5d042076e4a04" Oct 30 18:30:23.461447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.461422 5024 scope.go:122] "RemoveContainer" containerID="86d0715f0b745015d42c6ad5112cf8c75f01b955b17113ba9533dcc78871f6ad" Oct 30 18:30:23.461764 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:23.461742 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq_odh-ai-gateway-infra(f85b4cad-d13c-4257-b1f8-db84d996bea5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" Oct 30 18:30:23.501400 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:23.501347 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" podStartSLOduration=2.501332716 podStartE2EDuration="2.501332716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:23.499847249 +0000 UTC m=+1774.320559025" watchObservedRunningTime="2026-10-30 18:30:23.501332716 +0000 UTC m=+1774.322044492" Oct 30 18:30:24.180685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.180646 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:24.546557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.546480 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:24.701976 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.701946 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:24.855619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.855506 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\") pod \"f85b4cad-d13c-4257-b1f8-db84d996bea5\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " Oct 30 18:30:24.855619 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.855578 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\" (UniqueName: \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\") pod \"f85b4cad-d13c-4257-b1f8-db84d996bea5\" (UID: \"f85b4cad-d13c-4257-b1f8-db84d996bea5\") " Oct 30 18:30:24.855989 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.855964 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca" pod "f85b4cad-d13c-4257-b1f8-db84d996bea5" (UID: "f85b4cad-d13c-4257-b1f8-db84d996bea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:24.857900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.857877 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn" pod "f85b4cad-d13c-4257-b1f8-db84d996bea5" (UID: "f85b4cad-d13c-4257-b1f8-db84d996bea5"). InnerVolumeSpecName "kube-api-access-ddhnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:24.956771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.956736 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f85b4cad-d13c-4257-b1f8-db84d996bea5-maas-api-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:24.956771 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:24.956770 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddhnn\" (UniqueName: \"kubernetes.io/projected/f85b4cad-d13c-4257-b1f8-db84d996bea5-kube-api-access-ddhnn\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:25.475947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.475917 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" Oct 30 18:30:25.476138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.475958 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq" event={"ID":"f85b4cad-d13c-4257-b1f8-db84d996bea5","Type":"ContainerDied","Data":"b4d628601e2b62e79f235b95abea821089b55861c3a0b802bbf1c551916f2a06"} Oct 30 18:30:25.476138 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.476024 5024 scope.go:122] "RemoveContainer" containerID="86d0715f0b745015d42c6ad5112cf8c75f01b955b17113ba9533dcc78871f6ad" Oct 30 18:30:25.524634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.524584 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:25.528278 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.528252 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f0837a-29889750-8jlpq"] Oct 30 18:30:25.707078 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.707042 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:30:25.707911 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:25.707876 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" containerID="cri-o://5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928" gracePeriod=30 Oct 30 18:30:27.305966 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:27.305930 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" path="/var/lib/kubelet/pods/f85b4cad-d13c-4257-b1f8-db84d996bea5/volumes" Oct 30 18:30:30.982299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:30.982228 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:30:31.135511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135475 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135519 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135540 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135561 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135627 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135660 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.135774 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135700 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.136088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135781 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.136088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135818 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\") pod \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\" (UID: \"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6\") " Oct 30 18:30:31.136088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.135843 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.136088 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136005 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.136284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136114 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.136284 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136216 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.136363 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136323 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:31.136572 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136550 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.136658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136574 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.136658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136589 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.136658 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.136620 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.138131 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.138105 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.138249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.138141 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.138249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.138187 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "kube-api-access-j88w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.138508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.138490 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo" pod "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" (UID: "b1bc465e-81c5-4ee1-9d36-fdd60980f9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:31.237685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.237556 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j88w5\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-kube-api-access-j88w5\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.237685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.237590 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.237685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.237629 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.237685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.237638 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.237685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.237647 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.516249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.516151 5024 generic.go:356] "Generic (PLEG): container finished" podID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerID="5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928" exitCode=0 Oct 30 18:30:31.516249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.516229 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" Oct 30 18:30:31.516475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.516248 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" event={"ID":"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6","Type":"ContainerDied","Data":"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928"} Oct 30 18:30:31.516475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.516319 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r" event={"ID":"b1bc465e-81c5-4ee1-9d36-fdd60980f9e6","Type":"ContainerDied","Data":"da82aab9752dc41cfb6130f5561c62e452aaae77a4486d95cc80ca6f6674ff81"} Oct 30 18:30:31.516475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.516345 5024 scope.go:122] "RemoveContainer" containerID="5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928" Oct 30 18:30:31.527813 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.527796 5024 scope.go:122] "RemoveContainer" containerID="5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928" Oct 30 18:30:31.528132 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:31.528108 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928\": container with ID starting with 5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928 not found: ID does not exist" containerID="5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928" Oct 30 18:30:31.528184 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.528136 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928"} err="failed to get container status \"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928\": rpc error: code = NotFound desc = could not find container \"5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928\": container with ID starting with 5650540a35d7358d7053f900f801af198cca231d9c3386bcc16c142548d2c928 not found: ID does not exist" Oct 30 18:30:31.565246 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.565214 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:30:31.575579 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:31.575553 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-dz49r"] Oct 30 18:30:32.180382 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:32.180335 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:32.193373 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:32.193338 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:32.550734 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:32.550637 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:30:33.304704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:33.304662 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" path="/var/lib/kubelet/pods/b1bc465e-81c5-4ee1-9d36-fdd60980f9e6/volumes" Oct 30 18:30:40.455525 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:30:40.455484 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:30:50.772994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:50.772966 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:30:50.786305 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:50.786276 5024 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 18:30:50.796528 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:30:50.796502 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:31:10.455399 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:31:10.455357 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:31:40.458907 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:31:40.458809 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:31:55.783899 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:55.783852 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:31:55.784431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:55.784102 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerName="discovery" containerID="cri-o://2b0f299dfc139d73ac9776678b1f6da1fd89260c2733244fe821b7e94e39e2f4" gracePeriod=30 Oct 30 18:31:56.009089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.009049 5024 generic.go:356] "Generic (PLEG): container finished" podID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerID="2b0f299dfc139d73ac9776678b1f6da1fd89260c2733244fe821b7e94e39e2f4" exitCode=0 Oct 30 18:31:56.009289 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.009121 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" event={"ID":"1f5e8b1e-d5f1-48d7-94d1-442d974b7093","Type":"ContainerDied","Data":"2b0f299dfc139d73ac9776678b1f6da1fd89260c2733244fe821b7e94e39e2f4"} Oct 30 18:31:56.058685 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.058657 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:31:56.149037 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149002 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149237 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149046 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149237 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149119 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149237 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149151 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149237 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149212 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149314 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149379 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\") pod \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\" (UID: \"1f5e8b1e-d5f1-48d7-94d1-442d974b7093\") " Oct 30 18:31:56.149634 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149489 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:56.149908 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.149868 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-ca-configmap\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.151693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.151659 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:56.151926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.151898 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:56.152175 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.152145 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "kube-api-access-6m6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:56.152260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.152172 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:56.152260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.152205 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:56.152260 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.152244 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token" pod "1f5e8b1e-d5f1-48d7-94d1-442d974b7093" (UID: "1f5e8b1e-d5f1-48d7-94d1-442d974b7093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:56.251834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251795 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.251834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251827 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6wz\" (UniqueName: \"kubernetes.io/projected/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-kube-api-access-6m6wz\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.251834 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251838 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-csr-dns-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.252086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251847 5024 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-local-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.252086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251856 5024 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-istio-kubeconfig\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:56.252086 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:56.251864 5024 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f5e8b1e-d5f1-48d7-94d1-442d974b7093-cacerts\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.015884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.015854 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" Oct 30 18:31:57.016344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.015853 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf" event={"ID":"1f5e8b1e-d5f1-48d7-94d1-442d974b7093","Type":"ContainerDied","Data":"ee0061fc1367623d7ffd0fed9e16e542111832a8ad9bc3ec83d6b487263b671a"} Oct 30 18:31:57.016344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.015986 5024 scope.go:122] "RemoveContainer" containerID="2b0f299dfc139d73ac9776678b1f6da1fd89260c2733244fe821b7e94e39e2f4" Oct 30 18:31:57.076921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.076888 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:31:57.087480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.087450 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xttzf"] Oct 30 18:31:57.305299 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:31:57.305212 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" path="/var/lib/kubelet/pods/1f5e8b1e-d5f1-48d7-94d1-442d974b7093/volumes" Oct 30 18:32:07.289659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.289614 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:32:07.622010 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.621923 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l"] Oct 30 18:32:07.622404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622391 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" Oct 30 18:32:07.622446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622405 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" Oct 30 18:32:07.622446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622438 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622446 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622443 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622452 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerName="discovery" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622458 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerName="discovery" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622466 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622471 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622477 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622535 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622481 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622550 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bc465e-81c5-4ee1-9d36-fdd60980f9e6" containerName="istio-proxy" Oct 30 18:32:07.622764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622561 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622567 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5e8b1e-d5f1-48d7-94d1-442d974b7093" containerName="discovery" Oct 30 18:32:07.622764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622572 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.622764 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.622578 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85b4cad-d13c-4257-b1f8-db84d996bea5" containerName="cleanup" Oct 30 18:32:07.627676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.627657 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.650006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.649981 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:07.654390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.654367 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l"] Oct 30 18:32:07.665371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.665343 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:32:07.777897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.777853 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.778082 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.777920 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.778082 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.778000 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftwb\" (UniqueName: \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.879485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.879401 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.879485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.879475 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.879745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.879504 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftwb\" (UniqueName: \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.880236 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.880210 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.881952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.881931 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.897331 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.897299 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftwb\" (UniqueName: \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\") pod \"maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:07.941054 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:07.941022 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:08.240153 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:08.240125 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l"] Oct 30 18:32:08.246690 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:32:08.246617 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dda4751_2c6b_4522_9c76_9dba665a9b1e.slice/crio-3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e WatchSource:0}: Error finding container 3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e: Status 404 returned error can't find the container with id 3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e Oct 30 18:32:09.085521 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:09.085490 5024 generic.go:356] "Generic (PLEG): container finished" podID="6dda4751-2c6b-4522-9c76-9dba665a9b1e" containerID="4689706f704b975d99d82ddcbea04ece32b3da86336ef8765ee2bff0115f5e97" exitCode=0 Oct 30 18:32:09.085918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:09.085564 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" event={"ID":"6dda4751-2c6b-4522-9c76-9dba665a9b1e","Type":"ContainerDied","Data":"4689706f704b975d99d82ddcbea04ece32b3da86336ef8765ee2bff0115f5e97"} Oct 30 18:32:09.085918 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:09.085637 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" event={"ID":"6dda4751-2c6b-4522-9c76-9dba665a9b1e","Type":"ContainerStarted","Data":"3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e"} Oct 30 18:32:10.269089 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.269058 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:10.409023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.408997 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\" (UniqueName: \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\") pod \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " Oct 30 18:32:10.409188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.409078 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\") pod \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " Oct 30 18:32:10.409188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.409183 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\") pod \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\" (UID: \"6dda4751-2c6b-4522-9c76-9dba665a9b1e\") " Oct 30 18:32:10.409483 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.409456 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca" pod "6dda4751-2c6b-4522-9c76-9dba665a9b1e" (UID: "6dda4751-2c6b-4522-9c76-9dba665a9b1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:10.411361 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.411343 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb" pod "6dda4751-2c6b-4522-9c76-9dba665a9b1e" (UID: "6dda4751-2c6b-4522-9c76-9dba665a9b1e"). InnerVolumeSpecName "kube-api-access-sftwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:10.411438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.411359 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca" pod "6dda4751-2c6b-4522-9c76-9dba665a9b1e" (UID: "6dda4751-2c6b-4522-9c76-9dba665a9b1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:10.457562 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:32:10.457527 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:32:10.510624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.510560 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:10.510624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.510611 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dda4751-2c6b-4522-9c76-9dba665a9b1e-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:10.510624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:10.510625 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftwb\" (UniqueName: \"kubernetes.io/projected/6dda4751-2c6b-4522-9c76-9dba665a9b1e-kube-api-access-sftwb\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.099124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:11.099085 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" Oct 30 18:32:11.099124 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:11.099110 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l" event={"ID":"6dda4751-2c6b-4522-9c76-9dba665a9b1e","Type":"ContainerDied","Data":"3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e"} Oct 30 18:32:11.099344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:11.099165 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3baef86cb623d088b5a59d0a856b6106382baff0d2e50c0ee02970634d845c1e" Oct 30 18:32:19.274467 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:19.274432 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:32:22.311292 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.311234 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" podUID="598deb48-d868-41c4-a40f-b5909b99702d" containerName="main" containerID="cri-o://ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4" gracePeriod=45 Oct 30 18:32:22.577160 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.577096 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:32:22.634872 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.634826 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.634919 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.634989 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\" (UniqueName: \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635036 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635073 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635201 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\" (UniqueName: \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\") pod \"598deb48-d868-41c4-a40f-b5909b99702d\" (UID: \"598deb48-d868-41c4-a40f-b5909b99702d\") " Oct 30 18:32:22.635241 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635203 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.635416 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635362 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.635469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.635448 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.636169 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.636143 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.636307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.636173 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.636307 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.636187 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.637328 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.637299 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.637443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.637330 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:22.637507 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.637466 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq" pod "598deb48-d868-41c4-a40f-b5909b99702d" (UID: "598deb48-d868-41c4-a40f-b5909b99702d"). InnerVolumeSpecName "kube-api-access-52qsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:22.737422 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.737387 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598deb48-d868-41c4-a40f-b5909b99702d-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.737422 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.737419 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52qsq\" (UniqueName: \"kubernetes.io/projected/598deb48-d868-41c4-a40f-b5909b99702d-kube-api-access-52qsq\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.737422 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:22.737429 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598deb48-d868-41c4-a40f-b5909b99702d-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:23.172923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.172888 5024 generic.go:356] "Generic (PLEG): container finished" podID="598deb48-d868-41c4-a40f-b5909b99702d" containerID="ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4" exitCode=0 Oct 30 18:32:23.173114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.172940 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" Oct 30 18:32:23.173114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.172964 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" event={"ID":"598deb48-d868-41c4-a40f-b5909b99702d","Type":"ContainerDied","Data":"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4"} Oct 30 18:32:23.173114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.172995 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn" event={"ID":"598deb48-d868-41c4-a40f-b5909b99702d","Type":"ContainerDied","Data":"0dc83833d834ca9c6d0e7fe69b40b6c0a604a833288035042d284f3ef040cc22"} Oct 30 18:32:23.173114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.173011 5024 scope.go:122] "RemoveContainer" containerID="ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4" Oct 30 18:32:23.184447 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.184392 5024 scope.go:122] "RemoveContainer" containerID="ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4" Oct 30 18:32:23.184761 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:32:23.184733 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4\": container with ID starting with ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4 not found: ID does not exist" containerID="ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4" Oct 30 18:32:23.184841 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.184773 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4"} err="failed to get container status \"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4\": rpc error: code = NotFound desc = could not find container \"ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4\": container with ID starting with ca15cf46d71d4a0fd176829e09f97c74c95e35f1c5b6386bc54a3f5af54d7ed4 not found: ID does not exist" Oct 30 18:32:23.222193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.222154 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:32:23.232529 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.232503 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-69c44c/test-model-69c44c-kserve-6c64877c97-6jdvn"] Oct 30 18:32:23.304956 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:23.304919 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598deb48-d868-41c4-a40f-b5909b99702d" path="/var/lib/kubelet/pods/598deb48-d868-41c4-a40f-b5909b99702d/volumes" Oct 30 18:32:26.980702 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:26.980657 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l"] Oct 30 18:32:26.992897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:26.992864 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0f0c50-ndg9l"] Oct 30 18:32:27.305812 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:27.305731 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dda4751-2c6b-4522-9c76-9dba665a9b1e" path="/var/lib/kubelet/pods/6dda4751-2c6b-4522-9c76-9dba665a9b1e/volumes" Oct 30 18:32:34.295907 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.295855 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerName="main" containerID="cri-o://8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354" gracePeriod=45 Oct 30 18:32:34.571026 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.570998 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:32:34.671853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.671821 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.671907 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.671955 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672001 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672031 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672099 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\" (UniqueName: \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\") pod \"b63940fb-7c10-4316-b7f8-46532f3d849d\" (UID: \"b63940fb-7c10-4316-b7f8-46532f3d849d\") " Oct 30 18:32:34.672257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672124 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:34.672257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672209 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:34.672257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672247 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:34.672552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672456 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:34.672552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672481 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:34.672552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.672501 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:34.674578 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.674546 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:34.674924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.674901 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:34.675018 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.674922 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66" pod "b63940fb-7c10-4316-b7f8-46532f3d849d" (UID: "b63940fb-7c10-4316-b7f8-46532f3d849d"). InnerVolumeSpecName "kube-api-access-tfv66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:34.773226 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.773188 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63940fb-7c10-4316-b7f8-46532f3d849d-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:34.773391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.773239 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63940fb-7c10-4316-b7f8-46532f3d849d-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:34.773391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:34.773252 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfv66\" (UniqueName: \"kubernetes.io/projected/b63940fb-7c10-4316-b7f8-46532f3d849d-kube-api-access-tfv66\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.258205 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.258166 5024 generic.go:356] "Generic (PLEG): container finished" podID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerID="8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354" exitCode=0 Oct 30 18:32:35.258419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.258229 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" Oct 30 18:32:35.258419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.258250 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" event={"ID":"b63940fb-7c10-4316-b7f8-46532f3d849d","Type":"ContainerDied","Data":"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354"} Oct 30 18:32:35.258419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.258297 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" event={"ID":"b63940fb-7c10-4316-b7f8-46532f3d849d","Type":"ContainerDied","Data":"459b40a4c3f4f65a8021a0f43bd5ea92885ed6c662914361157db2650627e61a"} Oct 30 18:32:35.258419 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.258315 5024 scope.go:122] "RemoveContainer" containerID="8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354" Oct 30 18:32:35.268933 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.268916 5024 scope.go:122] "RemoveContainer" containerID="8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354" Oct 30 18:32:35.269210 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:32:35.269192 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354\": container with ID starting with 8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354 not found: ID does not exist" containerID="8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354" Oct 30 18:32:35.269282 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.269217 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354"} err="failed to get container status \"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354\": rpc error: code = NotFound desc = could not find container \"8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354\": container with ID starting with 8f7ad74ff0e8cf8df76c35dc6e638c5a689f92b4c014af2ebc8a33a3cdc4a354 not found: ID does not exist" Oct 30 18:32:35.308955 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.308921 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:32:35.318625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.318578 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49"] Oct 30 18:32:35.537656 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:35.537531 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c9a107/test-model-c9a107-kserve-56bb465b97-gnk49" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.130:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:32:37.304945 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:32:37.304907 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" path="/var/lib/kubelet/pods/b63940fb-7c10-4316-b7f8-46532f3d849d/volumes" Oct 30 18:32:40.456067 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:32:40.456026 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:33:10.457091 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:33:10.456986 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:33:11.231968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:11.231928 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:33:26.254122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.254059 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" containerID="cri-o://586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c" gracePeriod=45 Oct 30 18:33:26.547801 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.547773 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:33:26.556344 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.556314 5024 generic.go:356] "Generic (PLEG): container finished" podID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerID="586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c" exitCode=0 Oct 30 18:33:26.556498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.556379 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" Oct 30 18:33:26.556498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.556397 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" event={"ID":"0624902a-6c5d-41d0-a418-92a8b3b3baea","Type":"ContainerDied","Data":"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c"} Oct 30 18:33:26.556498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.556450 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" event={"ID":"0624902a-6c5d-41d0-a418-92a8b3b3baea","Type":"ContainerDied","Data":"3d813d293a3c420b6b53a9dfc3149fcc769db58e263fe20b671bf118f5a570a2"} Oct 30 18:33:26.556498 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.556473 5024 scope.go:122] "RemoveContainer" containerID="586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c" Oct 30 18:33:26.567211 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.567189 5024 scope.go:122] "RemoveContainer" containerID="586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c" Oct 30 18:33:26.567516 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:33:26.567494 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c\": container with ID starting with 586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c not found: ID does not exist" containerID="586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c" Oct 30 18:33:26.567714 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.567524 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c"} err="failed to get container status \"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c\": rpc error: code = NotFound desc = could not find container \"586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c\": container with ID starting with 586ceab50dcf31c64c92d1ec9640dbb64fdec7df1b322d42afe5f66049233d6c not found: ID does not exist" Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627099 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627176 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627212 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\" (UniqueName: \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627298 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627339 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.628189 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.627444 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\" (UniqueName: \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\") pod \"0624902a-6c5d-41d0-a418-92a8b3b3baea\" (UID: \"0624902a-6c5d-41d0-a418-92a8b3b3baea\") " Oct 30 18:33:26.629210 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.629170 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.629865 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.629833 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.633118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.630163 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.633755 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.633691 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:26.633915 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.633837 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.635028 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.634944 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff" pod "0624902a-6c5d-41d0-a418-92a8b3b3baea" (UID: "0624902a-6c5d-41d0-a418-92a8b3b3baea"). InnerVolumeSpecName "kube-api-access-hf5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:26.728975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.728933 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624902a-6c5d-41d0-a418-92a8b3b3baea-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.728975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.728967 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.728975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.728979 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.729217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.728995 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf5ff\" (UniqueName: \"kubernetes.io/projected/0624902a-6c5d-41d0-a418-92a8b3b3baea-kube-api-access-hf5ff\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.729217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.729007 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.729217 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.729018 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624902a-6c5d-41d0-a418-92a8b3b3baea-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.898762 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.898727 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:33:26.902057 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:26.902025 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz"] Oct 30 18:33:27.305036 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:27.305004 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" path="/var/lib/kubelet/pods/0624902a-6c5d-41d0-a418-92a8b3b3baea/volumes" Oct 30 18:33:27.521381 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:27.521333 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-w2-fde2c5-kserve-6fc7564b4b-phhrz" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 18:33:40.035927 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.035879 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:33:40.036512 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.036171 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" podUID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" containerName="maas-api" containerID="cri-o://f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2" gracePeriod=30 Oct 30 18:33:40.333257 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.333223 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:33:40.371478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.371439 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\") pod \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " Oct 30 18:33:40.371723 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.371634 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\") pod \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " Oct 30 18:33:40.371723 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.371690 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\" (UniqueName: \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\") pod \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " Oct 30 18:33:40.372335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.372066 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca" pod "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" (UID: "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:40.374436 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.374406 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7" pod "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" (UID: "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a"). InnerVolumeSpecName "kube-api-access-2ccb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:40.374568 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.374442 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs" pod "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" (UID: "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:40.458965 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:33:40.458914 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:33:40.472469 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.472421 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\") pod \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\" (UID: \"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a\") " Oct 30 18:33:40.472924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.472896 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:40.472924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.472926 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccb7\" (UniqueName: \"kubernetes.io/projected/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-kube-api-access-2ccb7\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:40.473140 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.472975 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:40.474978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.474945 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls" pod "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" (UID: "04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:40.574545 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.574455 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:40.637249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.637211 5024 generic.go:356] "Generic (PLEG): container finished" podID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" containerID="f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2" exitCode=0 Oct 30 18:33:40.637434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.637270 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" Oct 30 18:33:40.637434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.637289 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" event={"ID":"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a","Type":"ContainerDied","Data":"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2"} Oct 30 18:33:40.637434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.637368 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr" event={"ID":"04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a","Type":"ContainerDied","Data":"2a10b3b200a22b8777e87152a94dbba6ffe31b85255f345521be5d360958810c"} Oct 30 18:33:40.637434 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.637390 5024 scope.go:122] "RemoveContainer" containerID="f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2" Oct 30 18:33:40.648213 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.648190 5024 scope.go:122] "RemoveContainer" containerID="f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2" Oct 30 18:33:40.650505 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:33:40.650429 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2\": container with ID starting with f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2 not found: ID does not exist" containerID="f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2" Oct 30 18:33:40.650505 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.650479 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2"} err="failed to get container status \"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2\": rpc error: code = NotFound desc = could not find container \"f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2\": container with ID starting with f272aab9b5426042be6640f7843375b2c5a3cb037e2b02361ce0320ae261c8e2 not found: ID does not exist" Oct 30 18:33:40.697971 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.697935 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:33:40.708866 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:40.708826 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-f0837a-69946b7b46-hxxmr"] Oct 30 18:33:41.304781 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:41.304746 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" path="/var/lib/kubelet/pods/04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a/volumes" Oct 30 18:33:49.640234 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:49.640192 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:33:49.640884 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:49.640829 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" containerID="cri-o://232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd" gracePeriod=30 Oct 30 18:33:53.462909 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:53.462872 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:33:54.916176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:54.916109 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:33:55.028007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.027971 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028054 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028077 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028101 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028131 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028193 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028241 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028302 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028300 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028328 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.028385 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028356 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\") pod \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\" (UID: \"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208\") " Oct 30 18:33:55.028761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028569 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.028761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028635 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.028858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028778 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.028858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028802 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.028858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028816 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.028858 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028829 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.029066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.028948 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:55.030735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.030709 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "kube-api-access-dt58p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:55.030735 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.030732 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:55.030991 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.030970 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.031046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.031021 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token" pod "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" (UID: "d2a4e8e2-11bc-4e2e-b1e0-b833643f7208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:55.129482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.129449 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.129482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.129478 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.129482 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.129489 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.129769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.129498 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.129769 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.129507 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt58p\" (UniqueName: \"kubernetes.io/projected/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208-kube-api-access-dt58p\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.740261 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.740221 5024 generic.go:356] "Generic (PLEG): container finished" podID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerID="232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd" exitCode=0 Oct 30 18:33:55.740444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.740270 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" Oct 30 18:33:55.740444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.740309 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" event={"ID":"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208","Type":"ContainerDied","Data":"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd"} Oct 30 18:33:55.740444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.740363 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9" event={"ID":"d2a4e8e2-11bc-4e2e-b1e0-b833643f7208","Type":"ContainerDied","Data":"10f3971a9af66d896bded0b40d1aa8de8ab319153c358e1a4b3cec1899814a3b"} Oct 30 18:33:55.740444 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.740386 5024 scope.go:122] "RemoveContainer" containerID="232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd" Oct 30 18:33:55.750192 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.750162 5024 scope.go:122] "RemoveContainer" containerID="232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd" Oct 30 18:33:55.750474 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:33:55.750459 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd\": container with ID starting with 232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd not found: ID does not exist" containerID="232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd" Oct 30 18:33:55.750532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.750487 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd"} err="failed to get container status \"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd\": rpc error: code = NotFound desc = could not find container \"232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd\": container with ID starting with 232f07bde84c72e60dbe16823f302eaa4035716507d688a7c63603f63722e7fd not found: ID does not exist" Oct 30 18:33:55.798485 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.798438 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:33:55.809393 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:55.809361 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z7mp9"] Oct 30 18:33:57.304390 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:33:57.304356 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" path="/var/lib/kubelet/pods/d2a4e8e2-11bc-4e2e-b1e0-b833643f7208/volumes" Oct 30 18:34:08.484221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.484159 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" containerID="cri-o://ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc" gracePeriod=45 Oct 30 18:34:08.596166 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.596127 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 30 18:34:08.733418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.733392 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:34:08.756310 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756241 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756308 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756343 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756400 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756468 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756442 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756507 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\" (UniqueName: \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\") pod \"da269645-fa06-403f-a7e4-19169b6944a3\" (UID: \"da269645-fa06-403f-a7e4-19169b6944a3\") " Oct 30 18:34:08.756752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756548 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.756752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756646 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.756906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756870 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.756961 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756946 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-tmp-dir\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.757006 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.756968 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-model-cache\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.759007 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.758980 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:08.759130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.759030 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.759130 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.759042 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs" pod "da269645-fa06-403f-a7e4-19169b6944a3" (UID: "da269645-fa06-403f-a7e4-19169b6944a3"). InnerVolumeSpecName "kube-api-access-ggvcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:08.816232 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.816192 5024 generic.go:356] "Generic (PLEG): container finished" podID="da269645-fa06-403f-a7e4-19169b6944a3" containerID="ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc" exitCode=0 Oct 30 18:34:08.816435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.816278 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" Oct 30 18:34:08.816435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.816279 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" event={"ID":"da269645-fa06-403f-a7e4-19169b6944a3","Type":"ContainerDied","Data":"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc"} Oct 30 18:34:08.816435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.816409 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c" event={"ID":"da269645-fa06-403f-a7e4-19169b6944a3","Type":"ContainerDied","Data":"6855cb265d6a3e5890fadea84312f95b13beca5a7b80d3adacf9632d9866b089"} Oct 30 18:34:08.816435 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.816434 5024 scope.go:122] "RemoveContainer" containerID="ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc" Oct 30 18:34:08.827043 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.827022 5024 scope.go:122] "RemoveContainer" containerID="ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc" Oct 30 18:34:08.827666 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:34:08.827401 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc\": container with ID starting with ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc not found: ID does not exist" containerID="ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc" Oct 30 18:34:08.827666 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.827444 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc"} err="failed to get container status \"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc\": rpc error: code = NotFound desc = could not find container \"ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc\": container with ID starting with ae02292f297a9c464715175f58d91fa158576d8762483bad1767a530c79747cc not found: ID does not exist" Oct 30 18:34:08.857523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.857488 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da269645-fa06-403f-a7e4-19169b6944a3-tls-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.857523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.857518 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-dshm\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.857523 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.857527 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da269645-fa06-403f-a7e4-19169b6944a3-home\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.857850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.857535 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvcs\" (UniqueName: \"kubernetes.io/projected/da269645-fa06-403f-a7e4-19169b6944a3-kube-api-access-ggvcs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.866032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.865998 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:34:08.871139 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:08.871100 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-2-w4-4ef6ac-kserve-6f9d6557b5-6xr4c"] Oct 30 18:34:09.304784 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:09.304750 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da269645-fa06-403f-a7e4-19169b6944a3" path="/var/lib/kubelet/pods/da269645-fa06-403f-a7e4-19169b6944a3/volumes" Oct 30 18:34:10.455350 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:34:10.455317 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:34:14.422722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.422677 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:34:14.423347 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.422928 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" podUID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" containerName="maas-api" containerID="cri-o://fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd" gracePeriod=30 Oct 30 18:34:14.715386 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.715350 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:34:14.819713 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.819674 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\") pod \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " Oct 30 18:34:14.819921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.819740 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\") pod \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " Oct 30 18:34:14.819921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.819796 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\" (UniqueName: \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\") pod \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " Oct 30 18:34:14.819921 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.819876 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\") pod \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\" (UID: \"2b54f788-e7e4-439f-98b1-2c7ba3049db0\") " Oct 30 18:34:14.820110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.820040 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca" pod "2b54f788-e7e4-439f-98b1-2c7ba3049db0" (UID: "2b54f788-e7e4-439f-98b1-2c7ba3049db0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:14.820258 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.820234 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b54f788-e7e4-439f-98b1-2c7ba3049db0-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:14.822052 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.822028 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs" pod "2b54f788-e7e4-439f-98b1-2c7ba3049db0" (UID: "2b54f788-e7e4-439f-98b1-2c7ba3049db0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:14.822624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.822573 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv" pod "2b54f788-e7e4-439f-98b1-2c7ba3049db0" (UID: "2b54f788-e7e4-439f-98b1-2c7ba3049db0"). InnerVolumeSpecName "kube-api-access-scvzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:14.822624 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.822572 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls" pod "2b54f788-e7e4-439f-98b1-2c7ba3049db0" (UID: "2b54f788-e7e4-439f-98b1-2c7ba3049db0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:14.849882 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.849853 5024 generic.go:356] "Generic (PLEG): container finished" podID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" containerID="fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd" exitCode=0 Oct 30 18:34:14.850023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.849901 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" Oct 30 18:34:14.850023 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.849951 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" event={"ID":"2b54f788-e7e4-439f-98b1-2c7ba3049db0","Type":"ContainerDied","Data":"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd"} Oct 30 18:34:14.850133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.850059 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z" event={"ID":"2b54f788-e7e4-439f-98b1-2c7ba3049db0","Type":"ContainerDied","Data":"48c64f56316b415b04a8a0a27bc44431c42ac6663bc40afe6b1c25b936d5cde3"} Oct 30 18:34:14.850133 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.850093 5024 scope.go:122] "RemoveContainer" containerID="fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd" Oct 30 18:34:14.860239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.860219 5024 scope.go:122] "RemoveContainer" containerID="fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd" Oct 30 18:34:14.860500 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:34:14.860481 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd\": container with ID starting with fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd not found: ID does not exist" containerID="fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd" Oct 30 18:34:14.860573 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.860507 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd"} err="failed to get container status \"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd\": rpc error: code = NotFound desc = could not find container \"fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd\": container with ID starting with fcc9ff90d8b3b1f4fc73c27aa24d053977766f57ff74235b866514d282fb8acd not found: ID does not exist" Oct 30 18:34:14.917032 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.916999 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:34:14.920928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.920897 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:14.920928 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.920928 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvzv\" (UniqueName: \"kubernetes.io/projected/2b54f788-e7e4-439f-98b1-2c7ba3049db0-kube-api-access-scvzv\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:14.921146 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.920945 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b54f788-e7e4-439f-98b1-2c7ba3049db0-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:14.926704 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:14.926675 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ef6ac-6f54487786-bp46z"] Oct 30 18:34:15.305788 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:15.305749 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" path="/var/lib/kubelet/pods/2b54f788-e7e4-439f-98b1-2c7ba3049db0/volumes" Oct 30 18:34:22.779965 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:22.779912 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:34:22.781756 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:22.781715 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" containerID="cri-o://8e630cebf0637011ec974d177f7889e2c3f33f0c85d823ca6f159dbbe372251d" gracePeriod=30 Oct 30 18:34:27.928224 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:27.928189 5024 generic.go:356] "Generic (PLEG): container finished" podID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerID="8e630cebf0637011ec974d177f7889e2c3f33f0c85d823ca6f159dbbe372251d" exitCode=0 Oct 30 18:34:27.928660 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:27.928267 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" event={"ID":"6aac3789-e96f-4e9a-a95e-9fb527d0dee3","Type":"ContainerDied","Data":"8e630cebf0637011ec974d177f7889e2c3f33f0c85d823ca6f159dbbe372251d"} Oct 30 18:34:28.039470 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.039444 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:34:28.160873 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.160834 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.160882 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.160927 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.160969 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161010 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161039 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161081 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161068 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161113 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161254 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\") pod \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\" (UID: \"6aac3789-e96f-4e9a-a95e-9fb527d0dee3\") " Oct 30 18:34:28.161391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161270 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.161391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161255 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.161391 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161315 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.161676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161569 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-credential-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.161676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161588 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-socket\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.161676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161622 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-data\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.161850 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161754 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:28.161949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.161901 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.163364 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.163341 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.163460 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.163434 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:28.163962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.163936 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "kube-api-access-gzhbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:28.163962 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.163948 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo" pod "6aac3789-e96f-4e9a-a95e-9fb527d0dee3" (UID: "6aac3789-e96f-4e9a-a95e-9fb527d0dee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:28.262929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262890 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-podinfo\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.262929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262922 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-envoy\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.262929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262930 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istio-token\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.262929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262938 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-workload-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.263208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262946 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-istiod-ca-cert\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.263208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.262956 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhbq\" (UniqueName: \"kubernetes.io/projected/6aac3789-e96f-4e9a-a95e-9fb527d0dee3-kube-api-access-gzhbq\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.935136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.935104 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" Oct 30 18:34:28.935136 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.935122 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z" event={"ID":"6aac3789-e96f-4e9a-a95e-9fb527d0dee3","Type":"ContainerDied","Data":"151c6245e452c894e513d840513474b09ae783610abd6dabf55c7f4342b66412"} Oct 30 18:34:28.935659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.935179 5024 scope.go:122] "RemoveContainer" containerID="8e630cebf0637011ec974d177f7889e2c3f33f0c85d823ca6f159dbbe372251d" Oct 30 18:34:28.993793 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:28.993763 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:34:29.004694 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:29.004668 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-vpt5z"] Oct 30 18:34:29.306412 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:34:29.306325 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" path="/var/lib/kubelet/pods/6aac3789-e96f-4e9a-a95e-9fb527d0dee3/volumes" Oct 30 18:34:40.455443 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:34:40.455388 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:35:08.025186 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:35:08.025151 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j"] Oct 30 18:35:08.035553 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:35:08.035523 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d25855-l484j"] Oct 30 18:35:09.304715 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:35:09.304670 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83302028-01b9-46d5-83d1-a2f41dae05ec" path="/var/lib/kubelet/pods/83302028-01b9-46d5-83d1-a2f41dae05ec/volumes" Oct 30 18:35:10.455459 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:35:10.455418 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:35:40.454673 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:35:40.454628 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:35:50.809135 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:35:50.809107 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:35:50.829245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:35:50.829217 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:36:10.455215 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:36:10.455125 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:36:40.455468 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:36:40.455432 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:36:52.421053 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:36:52.421011 5024 scope.go:122] "RemoveContainer" containerID="6f4a6b69369de1e7fe8b0673f102f12564629d9bfe3091316558b2f033c0f6ae" Oct 30 18:37:10.454907 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:37:10.454868 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:37:40.457200 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:37:40.457101 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:38:10.455179 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:38:10.455140 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:38:40.457362 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:38:40.457325 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:38:52.448979 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:38:52.448941 5024 scope.go:122] "RemoveContainer" containerID="4689706f704b975d99d82ddcbea04ece32b3da86336ef8765ee2bff0115f5e97" Oct 30 18:39:10.463407 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:39:10.463292 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:39:40.460487 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:39:40.460448 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:40:10.454973 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:40:10.454934 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:40:40.465265 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:40:40.465180 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:40:50.843693 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:40:50.843665 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:40:50.860974 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:40:50.860951 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:41:10.454627 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:41:10.454563 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:41:40.458084 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:41:40.458023 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:42:10.454929 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:42:10.454839 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:42:40.455074 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:42:40.455028 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:43:10.455288 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:43:10.455238 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:43:40.455053 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:43:40.455012 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:44:10.455221 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:44:10.455182 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:44:40.455868 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:44:40.455827 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:45:10.455172 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:45:10.455083 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:45:14.664229 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664195 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt"] Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664608 5024 cpu_manager.go:414] "containerMap: removing container" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664620 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664636 5024 cpu_manager.go:414] "containerMap: removing container" podUID="598deb48-d868-41c4-a40f-b5909b99702d" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664641 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598deb48-d868-41c4-a40f-b5909b99702d" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664649 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6dda4751-2c6b-4522-9c76-9dba665a9b1e" containerName="revoke-keys" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664654 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dda4751-2c6b-4522-9c76-9dba665a9b1e" containerName="revoke-keys" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664661 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664666 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664673 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664678 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664685 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664689 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664704 5024 cpu_manager.go:414] "containerMap: removing container" podUID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" containerName="maas-api" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664709 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" containerName="maas-api" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664722 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" containerName="maas-api" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664729 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" containerName="maas-api" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664737 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" Oct 30 18:45:14.664745 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664742 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664822 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="da269645-fa06-403f-a7e4-19169b6944a3" containerName="main" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664832 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="598deb48-d868-41c4-a40f-b5909b99702d" containerName="main" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664839 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dda4751-2c6b-4522-9c76-9dba665a9b1e" containerName="revoke-keys" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664844 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a4e8e2-11bc-4e2e-b1e0-b833643f7208" containerName="istio-proxy" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664848 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63940fb-7c10-4316-b7f8-46532f3d849d" containerName="main" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664854 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0624902a-6c5d-41d0-a418-92a8b3b3baea" containerName="main" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664861 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bc7c22-41a1-46cb-b9ca-d8e5ec8c6d7a" containerName="maas-api" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664866 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b54f788-e7e4-439f-98b1-2c7ba3049db0" containerName="maas-api" Oct 30 18:45:14.665287 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.664872 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aac3789-e96f-4e9a-a95e-9fb527d0dee3" containerName="istio-proxy" Oct 30 18:45:14.667900 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.667883 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.683894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.683865 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt"] Oct 30 18:45:14.690181 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.690157 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:14.690355 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.690338 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:45:14.828441 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.828404 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.828667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.828449 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.828667 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.828588 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq4k\" (UniqueName: \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.929242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.929156 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.929242 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.929197 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.929437 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.929258 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq4k\" (UniqueName: \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.929978 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.929958 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.931875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.931856 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.946880 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.946858 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq4k\" (UniqueName: \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\") pod \"maas-api-revoke-sub-0271e108-ckbvt\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:14.981500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:14.981463 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:15.281475 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:15.281440 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt"] Oct 30 18:45:15.282551 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:45:15.282521 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f38b98_882f_4cce_9e3a_369a1f52b56a.slice/crio-208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971 WatchSource:0}: Error finding container 208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971: Status 404 returned error can't find the container with id 208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971 Oct 30 18:45:15.535424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:15.535333 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" event={"ID":"43f38b98-882f-4cce-9e3a-369a1f52b56a","Type":"ContainerStarted","Data":"208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971"} Oct 30 18:45:16.542929 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:16.542837 5024 generic.go:356] "Generic (PLEG): container finished" podID="43f38b98-882f-4cce-9e3a-369a1f52b56a" containerID="9df9fb27638a67fc2a08844dc0cb4b40baa10062764adca2dbec6fed22843627" exitCode=0 Oct 30 18:45:16.543306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:16.542917 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" event={"ID":"43f38b98-882f-4cce-9e3a-369a1f52b56a","Type":"ContainerDied","Data":"9df9fb27638a67fc2a08844dc0cb4b40baa10062764adca2dbec6fed22843627"} Oct 30 18:45:17.694677 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.694646 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:17.755855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.755818 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\") pod \"43f38b98-882f-4cce-9e3a-369a1f52b56a\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " Oct 30 18:45:17.755855 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.755862 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\" (UniqueName: \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\") pod \"43f38b98-882f-4cce-9e3a-369a1f52b56a\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " Oct 30 18:45:17.756110 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.755895 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\") pod \"43f38b98-882f-4cce-9e3a-369a1f52b56a\" (UID: \"43f38b98-882f-4cce-9e3a-369a1f52b56a\") " Oct 30 18:45:17.756268 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.756243 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca" pod "43f38b98-882f-4cce-9e3a-369a1f52b56a" (UID: "43f38b98-882f-4cce-9e3a-369a1f52b56a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:17.758238 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.758215 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k" pod "43f38b98-882f-4cce-9e3a-369a1f52b56a" (UID: "43f38b98-882f-4cce-9e3a-369a1f52b56a"). InnerVolumeSpecName "kube-api-access-bpq4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:17.758366 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.758333 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca" pod "43f38b98-882f-4cce-9e3a-369a1f52b56a" (UID: "43f38b98-882f-4cce-9e3a-369a1f52b56a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:17.856994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.856903 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:45:17.856994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.856937 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq4k\" (UniqueName: \"kubernetes.io/projected/43f38b98-882f-4cce-9e3a-369a1f52b56a-kube-api-access-bpq4k\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:45:17.856994 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:17.856951 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f38b98-882f-4cce-9e3a-369a1f52b56a-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.554707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:18.554673 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" Oct 30 18:45:18.554707 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:18.554699 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt" event={"ID":"43f38b98-882f-4cce-9e3a-369a1f52b56a","Type":"ContainerDied","Data":"208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971"} Oct 30 18:45:18.554938 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:18.554744 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208000d2acba6f82ba290c3278a5ad1897745422a49e209abb358fac0567b971" Oct 30 18:45:40.455832 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:45:40.455794 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:45:50.890943 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:50.890911 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:45:50.912035 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:45:50.912011 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:46:10.457081 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:46:10.457037 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:46:40.534429 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:46:40.534344 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:47:10.457182 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:47:10.457129 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:47:39.566343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:39.566305 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:47:39.566983 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:39.566523 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" podUID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" containerName="maas-api" containerID="cri-o://31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4" gracePeriod=30 Oct 30 18:47:40.456585 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:47:40.456538 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:47:40.834034 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.834006 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:47:40.994901 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.994808 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\") pod \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " Oct 30 18:47:40.995077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.994948 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\") pod \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " Oct 30 18:47:40.995077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.994975 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\" (UniqueName: \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\") pod \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " Oct 30 18:47:40.995077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.995002 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\") pod \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\" (UID: \"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6\") " Oct 30 18:47:40.995383 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.995346 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca" pod "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" (UID: "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:40.997115 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.997086 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs" pod "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" (UID: "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:40.997251 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.997227 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5" pod "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" (UID: "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6"). InnerVolumeSpecName "kube-api-access-84pp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:40.997301 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:40.997257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls" pod "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" (UID: "46aa2a0b-b5f3-4088-b564-bf1a2cd687c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:41.096689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.096647 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-metrics-certs\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:41.096689 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.096683 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-openshift-service-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:41.096894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.096700 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84pp5\" (UniqueName: \"kubernetes.io/projected/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-kube-api-access-84pp5\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:41.096894 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.096712 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6-maas-api-tls\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:41.419506 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.419471 5024 generic.go:356] "Generic (PLEG): container finished" podID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" containerID="31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4" exitCode=0 Oct 30 18:47:41.419760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.419512 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" Oct 30 18:47:41.419760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.419531 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" event={"ID":"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6","Type":"ContainerDied","Data":"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4"} Oct 30 18:47:41.419760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.419584 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5" event={"ID":"46aa2a0b-b5f3-4088-b564-bf1a2cd687c6","Type":"ContainerDied","Data":"6ccf101e25dad006732632e00ee01da0f68ca9e0b7cda6c4c013213c929c0532"} Oct 30 18:47:41.419760 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.419642 5024 scope.go:122] "RemoveContainer" containerID="31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4" Oct 30 18:47:41.429644 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.429612 5024 scope.go:122] "RemoveContainer" containerID="31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4" Oct 30 18:47:41.429972 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:47:41.429945 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4\": container with ID starting with 31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4 not found: ID does not exist" containerID="31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4" Oct 30 18:47:41.430046 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.429981 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4"} err="failed to get container status \"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4\": rpc error: code = NotFound desc = could not find container \"31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4\": container with ID starting with 31d4d11bd64239184d571677eeca0dea6c9ba482a246d69c359001c0775134c4 not found: ID does not exist" Oct 30 18:47:41.480384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.480351 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:47:41.491094 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:41.491056 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-dl8h5"] Oct 30 18:47:43.305012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:43.304976 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" path="/var/lib/kubelet/pods/46aa2a0b-b5f3-4088-b564-bf1a2cd687c6/volumes" Oct 30 18:47:51.276380 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276331 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br"] Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276742 5024 cpu_manager.go:414] "containerMap: removing container" podUID="43f38b98-882f-4cce-9e3a-369a1f52b56a" containerName="revoke-keys" Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276754 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f38b98-882f-4cce-9e3a-369a1f52b56a" containerName="revoke-keys" Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276792 5024 cpu_manager.go:414] "containerMap: removing container" podUID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" containerName="maas-api" Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276797 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" containerName="maas-api" Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276858 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f38b98-882f-4cce-9e3a-369a1f52b56a" containerName="revoke-keys" Oct 30 18:47:51.276897 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.276873 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="46aa2a0b-b5f3-4088-b564-bf1a2cd687c6" containerName="maas-api" Oct 30 18:47:51.279926 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.279905 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.304443 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.304408 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:47:51.304879 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.304858 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z7qv\"" Oct 30 18:47:51.310077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.310053 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:47:51.310343 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.310326 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:47:51.311113 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.311081 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br"] Oct 30 18:47:51.392622 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.392564 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.392813 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.392682 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmzj\" (UniqueName: \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.392861 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.392830 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.493875 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.493835 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.494063 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.493911 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.494107 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.494078 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmzj\" (UniqueName: \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.494511 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.494480 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.496384 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.496366 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.511871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.511838 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmzj\" (UniqueName: \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\") pod \"maas-api-revoke-keys-e2e-ait-069c47a6-b64br\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.592820 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.592725 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:51.888659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:51.888630 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br"] Oct 30 18:47:51.889758 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:47:51.889733 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c5c1c7_0161_4393_9a2d_907fe3bb7c37.slice/crio-9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c WatchSource:0}: Error finding container 9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c: Status 404 returned error can't find the container with id 9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c Oct 30 18:47:52.481431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:52.481329 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" event={"ID":"97c5c1c7-0161-4393-9a2d-907fe3bb7c37","Type":"ContainerStarted","Data":"34d01088c294bd2077b783076ebbe044ad6b557f4ea30f78c6d29f957e2c4d4b"} Oct 30 18:47:52.481431 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:52.481398 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" event={"ID":"97c5c1c7-0161-4393-9a2d-907fe3bb7c37","Type":"ContainerStarted","Data":"9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c"} Oct 30 18:47:52.529179 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:52.529122 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" podStartSLOduration=1.5291057590000001 podStartE2EDuration="1.529105759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:52.515321773 +0000 UTC m=+2823.336033552" watchObservedRunningTime="2026-10-30 18:47:52.529105759 +0000 UTC m=+2823.349817533" Oct 30 18:47:53.487754 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:53.487718 5024 generic.go:356] "Generic (PLEG): container finished" podID="97c5c1c7-0161-4393-9a2d-907fe3bb7c37" containerID="34d01088c294bd2077b783076ebbe044ad6b557f4ea30f78c6d29f957e2c4d4b" exitCode=0 Oct 30 18:47:53.488176 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:53.487787 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" event={"ID":"97c5c1c7-0161-4393-9a2d-907fe3bb7c37","Type":"ContainerDied","Data":"34d01088c294bd2077b783076ebbe044ad6b557f4ea30f78c6d29f957e2c4d4b"} Oct 30 18:47:54.633355 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.633325 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:54.827068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.826975 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\") pod \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " Oct 30 18:47:54.827068 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.827022 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\" (UniqueName: \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\") pod \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " Oct 30 18:47:54.827277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.827069 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\") pod \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\" (UID: \"97c5c1c7-0161-4393-9a2d-907fe3bb7c37\") " Oct 30 18:47:54.827406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.827386 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca" pod "97c5c1c7-0161-4393-9a2d-907fe3bb7c37" (UID: "97c5c1c7-0161-4393-9a2d-907fe3bb7c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:54.829338 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.829310 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca" pod "97c5c1c7-0161-4393-9a2d-907fe3bb7c37" (UID: "97c5c1c7-0161-4393-9a2d-907fe3bb7c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:54.829440 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.829383 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj" pod "97c5c1c7-0161-4393-9a2d-907fe3bb7c37" (UID: "97c5c1c7-0161-4393-9a2d-907fe3bb7c37"). InnerVolumeSpecName "kube-api-access-jhmzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:54.928117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.928063 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:54.928117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.928112 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:54.928117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:54.928126 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmzj\" (UniqueName: \"kubernetes.io/projected/97c5c1c7-0161-4393-9a2d-907fe3bb7c37-kube-api-access-jhmzj\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:47:55.499807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:55.499775 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" Oct 30 18:47:55.499807 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:55.499801 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br" event={"ID":"97c5c1c7-0161-4393-9a2d-907fe3bb7c37","Type":"ContainerDied","Data":"9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c"} Oct 30 18:47:55.500025 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:47:55.499845 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afabdcf70efe40c700d4451f6450779c765a4e35234ce0fd491a5e3b6c6101c" Oct 30 18:48:03.271522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:48:03.271420 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br"] Oct 30 18:48:03.282235 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:48:03.282195 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-069c47a6-b64br"] Oct 30 18:48:03.305559 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:48:03.305521 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c5c1c7-0161-4393-9a2d-907fe3bb7c37" path="/var/lib/kubelet/pods/97c5c1c7-0161-4393-9a2d-907fe3bb7c37/volumes" Oct 30 18:48:10.454355 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:48:10.454311 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:48:40.455190 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:48:40.455147 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:49:10.454655 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:49:10.454583 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:49:40.453634 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:49:40.453533 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:49:47.465905 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.465870 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx"] Oct 30 18:49:47.466372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.466247 5024 cpu_manager.go:414] "containerMap: removing container" podUID="97c5c1c7-0161-4393-9a2d-907fe3bb7c37" containerName="revoke-keys" Oct 30 18:49:47.466372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.466258 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c5c1c7-0161-4393-9a2d-907fe3bb7c37" containerName="revoke-keys" Oct 30 18:49:47.466372 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.466349 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c5c1c7-0161-4393-9a2d-907fe3bb7c37" containerName="revoke-keys" Oct 30 18:49:47.469297 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.469279 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.483574 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.483519 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:47.494448 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.494420 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z7qv\"" Oct 30 18:49:47.494556 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.494458 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:49:47.498484 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.498461 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx"] Oct 30 18:49:47.584012 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.583981 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:49:47.607552 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.607522 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.607699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.607568 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqlw\" (UniqueName: \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.607699 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.607685 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.709013 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.708973 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.709188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.709095 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.709188 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.709125 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqlw\" (UniqueName: \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.709836 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.709769 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.711571 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.711550 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.732353 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.732289 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqlw\" (UniqueName: \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\") pod \"maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:47.783193 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:47.783162 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:48.102577 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:48.102552 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx"] Oct 30 18:49:48.103651 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:49:48.103629 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11b1b16_353d_44df_869b_dca9b2519f1e.slice/crio-4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d WatchSource:0}: Error finding container 4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d: Status 404 returned error can't find the container with id 4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d Oct 30 18:49:49.090221 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:49.090182 5024 generic.go:356] "Generic (PLEG): container finished" podID="c11b1b16-353d-44df-869b-dca9b2519f1e" containerID="f9906e8e378bb87de1987b26d0ccc50ece47e48f9a79900535a441aa3346c6ef" exitCode=0 Oct 30 18:49:49.090641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:49.090251 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" event={"ID":"c11b1b16-353d-44df-869b-dca9b2519f1e","Type":"ContainerDied","Data":"f9906e8e378bb87de1987b26d0ccc50ece47e48f9a79900535a441aa3346c6ef"} Oct 30 18:49:49.090641 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:49.090335 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" event={"ID":"c11b1b16-353d-44df-869b-dca9b2519f1e","Type":"ContainerStarted","Data":"4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d"} Oct 30 18:49:50.229779 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.229758 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:50.333871 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.333841 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\") pod \"c11b1b16-353d-44df-869b-dca9b2519f1e\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " Oct 30 18:49:50.334055 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.333952 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\") pod \"c11b1b16-353d-44df-869b-dca9b2519f1e\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " Oct 30 18:49:50.334055 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.333995 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\" (UniqueName: \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\") pod \"c11b1b16-353d-44df-869b-dca9b2519f1e\" (UID: \"c11b1b16-353d-44df-869b-dca9b2519f1e\") " Oct 30 18:49:50.334283 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.334257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca" pod "c11b1b16-353d-44df-869b-dca9b2519f1e" (UID: "c11b1b16-353d-44df-869b-dca9b2519f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:50.336208 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.336178 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw" pod "c11b1b16-353d-44df-869b-dca9b2519f1e" (UID: "c11b1b16-353d-44df-869b-dca9b2519f1e"). InnerVolumeSpecName "kube-api-access-9wqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:50.336306 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.336272 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca" pod "c11b1b16-353d-44df-869b-dca9b2519f1e" (UID: "c11b1b16-353d-44df-869b-dca9b2519f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:50.434970 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.434942 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:49:50.435148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.435064 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqlw\" (UniqueName: \"kubernetes.io/projected/c11b1b16-353d-44df-869b-dca9b2519f1e-kube-api-access-9wqlw\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:49:50.435148 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:50.435085 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c11b1b16-353d-44df-869b-dca9b2519f1e-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:49:51.102503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:51.102469 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" Oct 30 18:49:51.102716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:51.102504 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx" event={"ID":"c11b1b16-353d-44df-869b-dca9b2519f1e","Type":"ContainerDied","Data":"4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d"} Oct 30 18:49:51.102716 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:49:51.102556 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbd7c226faf1c9cc3cc4608a3b689c9e973953a73f8232bfc68b35770ef8f2d" Oct 30 18:50:06.607095 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:06.607061 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx"] Oct 30 18:50:06.616297 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:06.616271 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-783444-r8dzx"] Oct 30 18:50:07.303849 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:07.303816 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c11b1b16-353d-44df-869b-dca9b2519f1e" path="/var/lib/kubelet/pods/c11b1b16-353d-44df-869b-dca9b2519f1e/volumes" Oct 30 18:50:10.454175 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:50:10.454133 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:50:18.034751 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:18.034710 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt"] Oct 30 18:50:18.045459 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:18.045422 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0271e108-ckbvt"] Oct 30 18:50:19.321090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:19.321054 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f38b98-882f-4cce-9e3a-369a1f52b56a" path="/var/lib/kubelet/pods/43f38b98-882f-4cce-9e3a-369a1f52b56a/volumes" Oct 30 18:50:31.165827 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.165785 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9"] Oct 30 18:50:31.166239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.166206 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c11b1b16-353d-44df-869b-dca9b2519f1e" containerName="revoke-keys" Oct 30 18:50:31.166239 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.166219 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c11b1b16-353d-44df-869b-dca9b2519f1e" containerName="revoke-keys" Oct 30 18:50:31.166328 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.166317 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c11b1b16-353d-44df-869b-dca9b2519f1e" containerName="revoke-keys" Oct 30 18:50:31.169765 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.169746 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.185487 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.185453 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9"] Oct 30 18:50:31.192404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.192372 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:50:31.192404 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.192382 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:50:31.192636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.192410 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:50:31.192636 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.192513 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z7qv\"" Oct 30 18:50:31.260968 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.260926 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.261171 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.261008 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwx68\" (UniqueName: \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.261171 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.261068 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.361722 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.361682 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.361936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.361739 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwx68\" (UniqueName: \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.361936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.361777 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.362430 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.362410 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.364311 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.364286 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.381200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.381163 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwx68\" (UniqueName: \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\") pod \"maas-api-revoke-sub-f03629db-rq4k9\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.484406 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.484322 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:31.784550 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:31.784518 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9"] Oct 30 18:50:31.785878 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:50:31.785848 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf214bd63_fb34_4c8d_a290_1d83088302c0.slice/crio-0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80 WatchSource:0}: Error finding container 0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80: Status 404 returned error can't find the container with id 0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80 Oct 30 18:50:32.348610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:32.348494 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" event={"ID":"f214bd63-fb34-4c8d-a290-1d83088302c0","Type":"ContainerStarted","Data":"021fdfde5c483e690fed7069c70c746ffb1c8238576248ef97f137070caff3d9"} Oct 30 18:50:32.348610 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:32.348563 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" event={"ID":"f214bd63-fb34-4c8d-a290-1d83088302c0","Type":"ContainerStarted","Data":"0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80"} Oct 30 18:50:32.386255 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:32.386194 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" podStartSLOduration=1.386179947 podStartE2EDuration="1.386179947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:32.384961626 +0000 UTC m=+2983.205673420" watchObservedRunningTime="2026-10-30 18:50:32.386179947 +0000 UTC m=+2983.206891722" Oct 30 18:50:33.355315 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:33.355275 5024 generic.go:356] "Generic (PLEG): container finished" podID="f214bd63-fb34-4c8d-a290-1d83088302c0" containerID="021fdfde5c483e690fed7069c70c746ffb1c8238576248ef97f137070caff3d9" exitCode=0 Oct 30 18:50:33.355748 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:33.355363 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" event={"ID":"f214bd63-fb34-4c8d-a290-1d83088302c0","Type":"ContainerDied","Data":"021fdfde5c483e690fed7069c70c746ffb1c8238576248ef97f137070caff3d9"} Oct 30 18:50:34.507565 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.507534 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:34.594547 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.594499 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\") pod \"f214bd63-fb34-4c8d-a290-1d83088302c0\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " Oct 30 18:50:34.594809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.594659 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\") pod \"f214bd63-fb34-4c8d-a290-1d83088302c0\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " Oct 30 18:50:34.594809 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.594694 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\" (UniqueName: \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\") pod \"f214bd63-fb34-4c8d-a290-1d83088302c0\" (UID: \"f214bd63-fb34-4c8d-a290-1d83088302c0\") " Oct 30 18:50:34.594973 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.594936 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca" pod "f214bd63-fb34-4c8d-a290-1d83088302c0" (UID: "f214bd63-fb34-4c8d-a290-1d83088302c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:34.595098 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.595046 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:34.596952 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.596923 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca" pod "f214bd63-fb34-4c8d-a290-1d83088302c0" (UID: "f214bd63-fb34-4c8d-a290-1d83088302c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:34.597059 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.596985 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68" pod "f214bd63-fb34-4c8d-a290-1d83088302c0" (UID: "f214bd63-fb34-4c8d-a290-1d83088302c0"). InnerVolumeSpecName "kube-api-access-vwx68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:34.695614 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.695560 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwx68\" (UniqueName: \"kubernetes.io/projected/f214bd63-fb34-4c8d-a290-1d83088302c0-kube-api-access-vwx68\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:34.695815 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:34.695646 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f214bd63-fb34-4c8d-a290-1d83088302c0-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:35.366831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:35.366800 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" Oct 30 18:50:35.366831 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:35.366811 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-rq4k9" event={"ID":"f214bd63-fb34-4c8d-a290-1d83088302c0","Type":"ContainerDied","Data":"0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80"} Oct 30 18:50:35.367109 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:35.366888 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c3314e47232c307303db8ce950f85945cb900181988504b4915614e5233cd80" Oct 30 18:50:40.454520 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:50:40.454479 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:50:47.075696 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.068549 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm"] Oct 30 18:50:47.080196 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.080158 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f214bd63-fb34-4c8d-a290-1d83088302c0" containerName="revoke-keys" Oct 30 18:50:47.080452 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.080431 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f214bd63-fb34-4c8d-a290-1d83088302c0" containerName="revoke-keys" Oct 30 18:50:47.080717 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.080699 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f214bd63-fb34-4c8d-a290-1d83088302c0" containerName="revoke-keys" Oct 30 18:50:47.132532 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.132495 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm"] Oct 30 18:50:47.132996 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.132904 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.176626 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.163752 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.227969 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.228188 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z7qv\"" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.228240 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.228705 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9q5b\" (UniqueName: \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.228784 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.230625 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.228825 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.330200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.330102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.330200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.330146 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.330503 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.330243 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9q5b\" (UniqueName: \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.330947 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.330923 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.332708 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.332688 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.410522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.410484 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9q5b\" (UniqueName: \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\") pod \"maas-api-revoke-sub-0165d7d2-hfmjm\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.532200 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.532150 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:47.920117 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:47.920074 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm"] Oct 30 18:50:47.921219 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:50:47.920631 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a464a7e_e124_4444_ab04_db251826dabb.slice/crio-a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd WatchSource:0}: Error finding container a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd: Status 404 returned error can't find the container with id a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd Oct 30 18:50:48.438090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:48.437987 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" event={"ID":"8a464a7e-e124-4444-ab04-db251826dabb","Type":"ContainerStarted","Data":"8eb4efee4fe01d84fe9bf31ee277ed03c9a4fa281291c6605da01918c2f21d21"} Oct 30 18:50:48.438090 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:48.438063 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" event={"ID":"8a464a7e-e124-4444-ab04-db251826dabb","Type":"ContainerStarted","Data":"a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd"} Oct 30 18:50:48.517478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:48.517396 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" podStartSLOduration=1.5173714820000002 podStartE2EDuration="1.517371482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:48.502155232 +0000 UTC m=+2999.322867009" watchObservedRunningTime="2026-10-30 18:50:48.517371482 +0000 UTC m=+2999.338083258" Oct 30 18:50:49.444902 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:49.444862 5024 generic.go:356] "Generic (PLEG): container finished" podID="8a464a7e-e124-4444-ab04-db251826dabb" containerID="8eb4efee4fe01d84fe9bf31ee277ed03c9a4fa281291c6605da01918c2f21d21" exitCode=0 Oct 30 18:50:49.445424 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:49.444951 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" event={"ID":"8a464a7e-e124-4444-ab04-db251826dabb","Type":"ContainerDied","Data":"8eb4efee4fe01d84fe9bf31ee277ed03c9a4fa281291c6605da01918c2f21d21"} Oct 30 18:50:50.603701 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.603671 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:50.656191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.656156 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\") pod \"8a464a7e-e124-4444-ab04-db251826dabb\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " Oct 30 18:50:50.656371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.656264 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\" (UniqueName: \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\") pod \"8a464a7e-e124-4444-ab04-db251826dabb\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " Oct 30 18:50:50.656371 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.656334 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\") pod \"8a464a7e-e124-4444-ab04-db251826dabb\" (UID: \"8a464a7e-e124-4444-ab04-db251826dabb\") " Oct 30 18:50:50.656663 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.656576 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca" pod "8a464a7e-e124-4444-ab04-db251826dabb" (UID: "8a464a7e-e124-4444-ab04-db251826dabb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:50.656777 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.656675 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:50.658785 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.658741 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca" pod "8a464a7e-e124-4444-ab04-db251826dabb" (UID: "8a464a7e-e124-4444-ab04-db251826dabb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:50.658906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.658852 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b" pod "8a464a7e-e124-4444-ab04-db251826dabb" (UID: "8a464a7e-e124-4444-ab04-db251826dabb"). InnerVolumeSpecName "kube-api-access-b9q5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:50.757705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.757565 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9q5b\" (UniqueName: \"kubernetes.io/projected/8a464a7e-e124-4444-ab04-db251826dabb-kube-api-access-b9q5b\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:50.757705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.757644 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a464a7e-e124-4444-ab04-db251826dabb-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:50:50.934582 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.934553 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:50:50.947640 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:50.947562 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:50:51.456721 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:51.456689 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" Oct 30 18:50:51.456949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:51.456719 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfmjm" event={"ID":"8a464a7e-e124-4444-ab04-db251826dabb","Type":"ContainerDied","Data":"a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd"} Oct 30 18:50:51.456949 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:51.456774 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26b58cb0fc4133f54d3dc772a484acea553d15b2a4ff86925cd589e105446bd" Oct 30 18:50:57.975421 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.975382 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn"] Oct 30 18:50:57.975917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.975843 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8a464a7e-e124-4444-ab04-db251826dabb" containerName="revoke-keys" Oct 30 18:50:57.975917 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.975856 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a464a7e-e124-4444-ab04-db251826dabb" containerName="revoke-keys" Oct 30 18:50:57.976066 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.975942 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a464a7e-e124-4444-ab04-db251826dabb" containerName="revoke-keys" Oct 30 18:50:57.979490 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.979464 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:57.994975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:57.994944 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn"] Oct 30 18:50:58.001864 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.001832 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:50:58.002134 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.002097 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:50:58.003309 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.003287 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:50:58.003418 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.003323 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z7qv\"" Oct 30 18:50:58.022791 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.022740 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.022986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.022887 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.023100 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.023072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7thl\" (UniqueName: \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.124427 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.124391 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.124629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.124455 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7thl\" (UniqueName: \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.124629 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.124496 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.125112 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.125085 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.127021 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.126996 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.142177 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.142149 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7thl\" (UniqueName: \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\") pod \"maas-api-revoke-sub-6b19338f-qqmtn\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.293014 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.292924 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:50:58.591652 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:58.591622 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn"] Oct 30 18:50:58.592205 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:50:58.592176 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fa4061_0c90_4c4c_bfc6_137110ca60ef.slice/crio-180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4 WatchSource:0}: Error finding container 180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4: Status 404 returned error can't find the container with id 180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4 Oct 30 18:50:59.508611 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:59.508558 5024 generic.go:356] "Generic (PLEG): container finished" podID="a0fa4061-0c90-4c4c-bfc6-137110ca60ef" containerID="9e0f75fd5b7aac99065e4a8c7d6895690e9713b31e1ce89f32535a02960f4c13" exitCode=0 Oct 30 18:50:59.509105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:59.508638 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" event={"ID":"a0fa4061-0c90-4c4c-bfc6-137110ca60ef","Type":"ContainerDied","Data":"9e0f75fd5b7aac99065e4a8c7d6895690e9713b31e1ce89f32535a02960f4c13"} Oct 30 18:50:59.509105 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:50:59.508689 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" event={"ID":"a0fa4061-0c90-4c4c-bfc6-137110ca60ef","Type":"ContainerStarted","Data":"180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4"} Oct 30 18:51:00.681277 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.681245 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:51:00.751709 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.751670 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\") pod \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " Oct 30 18:51:00.751924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.751740 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\") pod \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " Oct 30 18:51:00.751924 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.751856 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\" (UniqueName: \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\") pod \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\" (UID: \"a0fa4061-0c90-4c4c-bfc6-137110ca60ef\") " Oct 30 18:51:00.752084 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.752061 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca" pod "a0fa4061-0c90-4c4c-bfc6-137110ca60ef" (UID: "a0fa4061-0c90-4c4c-bfc6-137110ca60ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:00.752295 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.752273 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-openshift-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:51:00.754719 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.754689 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca" pod "a0fa4061-0c90-4c4c-bfc6-137110ca60ef" (UID: "a0fa4061-0c90-4c4c-bfc6-137110ca60ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:00.759114 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.759084 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl" pod "a0fa4061-0c90-4c4c-bfc6-137110ca60ef" (UID: "a0fa4061-0c90-4c4c-bfc6-137110ca60ef"). InnerVolumeSpecName "kube-api-access-k7thl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:00.853835 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.853741 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-maas-api-xks-ca\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:51:00.853835 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:00.853775 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7thl\" (UniqueName: \"kubernetes.io/projected/a0fa4061-0c90-4c4c-bfc6-137110ca60ef-kube-api-access-k7thl\") on node \"ip-10-0-143-94.ec2.internal\" DevicePath \"\"" Oct 30 18:51:01.520473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:01.520441 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" Oct 30 18:51:01.520473 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:01.520452 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqmtn" event={"ID":"a0fa4061-0c90-4c4c-bfc6-137110ca60ef","Type":"ContainerDied","Data":"180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4"} Oct 30 18:51:01.520712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:01.520497 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="180cb8e50ed86e370e77b5783ea2e79ecd26305f5993e34d16b4f6ea2fbe0cd4" Oct 30 18:51:10.536308 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:51:10.536269 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:51:20.262294 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:20.262257 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-db7c8_cef5aee3-add8-449c-9635-58897b7023a9/manager/0.log" Oct 30 18:51:20.470033 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:20.469996 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7prn9_882bc53c-8e28-44f5-8309-da5cd21ff4b5/manager/0.log" Oct 30 18:51:20.696126 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:20.696092 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsmrg_18ef773b-33c1-4db5-b541-ecee76e8bba3/manager/0.log" Oct 30 18:51:21.678048 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:21.678017 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vgw6s_4536ebbb-f8ee-4a78-a0ee-d046bb448e16/proxy/0.log" Oct 30 18:51:22.810712 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:22.810681 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hfmjm_8a464a7e-e124-4444-ab04-db251826dabb/revoke-keys/0.log" Oct 30 18:51:23.549150 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:23.549119 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-qqmtn_a0fa4061-0c90-4c4c-bfc6-137110ca60ef/revoke-keys/0.log" Oct 30 18:51:23.742394 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:23.742359 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rq4k9_f214bd63-fb34-4c8d-a290-1d83088302c0/revoke-keys/0.log" Oct 30 18:51:26.084585 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:26.084554 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h8nms_28da65aa-8ccc-46c5-a8dc-06c341aa4e1c/kuadrant-console-plugin/0.log" Oct 30 18:51:28.301129 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:28.301095 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-zsxmb_3552bb13-3271-4be5-8d76-5bac03ce89df/kube-auth-proxy/0.log" Oct 30 18:51:29.079452 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:29.079416 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hn66q_29abc014-6eb3-474b-95b4-cc26370fd325/istio-proxy/0.log" Oct 30 18:51:29.454043 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:29.454006 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n789k_8029cccf-1a8c-431e-a244-c840c94bb846/istio-proxy/0.log" Oct 30 18:51:29.824413 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:29.824318 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vftgt_9475a6a1-002e-49b3-bb80-67db243d4e0f/istio-proxy/0.log" Oct 30 18:51:30.942522 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:30.942489 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log" Oct 30 18:51:40.454859 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:51:40.454813 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:51:43.441988 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:43.441954 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-db7c8_cef5aee3-add8-449c-9635-58897b7023a9/manager/0.log" Oct 30 18:51:43.646478 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:43.646443 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-7prn9_882bc53c-8e28-44f5-8309-da5cd21ff4b5/manager/0.log" Oct 30 18:51:43.861560 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:43.861479 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rsmrg_18ef773b-33c1-4db5-b541-ecee76e8bba3/manager/0.log" Oct 30 18:51:44.732524 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:44.732493 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vgw6s_4536ebbb-f8ee-4a78-a0ee-d046bb448e16/proxy/0.log" Oct 30 18:51:45.665398 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:45.665366 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hfmjm_8a464a7e-e124-4444-ab04-db251826dabb/revoke-keys/0.log" Oct 30 18:51:46.223497 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:46.223464 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-qqmtn_a0fa4061-0c90-4c4c-bfc6-137110ca60ef/revoke-keys/0.log" Oct 30 18:51:46.404761 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:46.404728 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-rq4k9_f214bd63-fb34-4c8d-a290-1d83088302c0/revoke-keys/0.log" Oct 30 18:51:48.392330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:48.392301 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h8nms_28da65aa-8ccc-46c5-a8dc-06c341aa4e1c/kuadrant-console-plugin/0.log" Oct 30 18:51:50.121799 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:50.121762 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-zsxmb_3552bb13-3271-4be5-8d76-5bac03ce89df/kube-auth-proxy/0.log" Oct 30 18:51:50.897500 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:50.897467 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hn66q_29abc014-6eb3-474b-95b4-cc26370fd325/istio-proxy/0.log" Oct 30 18:51:51.302543 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:51.302452 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n789k_8029cccf-1a8c-431e-a244-c840c94bb846/istio-proxy/0.log" Oct 30 18:51:51.686438 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:51.686410 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vftgt_9475a6a1-002e-49b3-bb80-67db243d4e0f/istio-proxy/0.log" Oct 30 18:51:52.571465 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:52.571429 5024 scope.go:122] "RemoveContainer" containerID="9df9fb27638a67fc2a08844dc0cb4b40baa10062764adca2dbec6fed22843627" Oct 30 18:51:52.851557 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:51:52.851469 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log" Oct 30 18:52:10.453813 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:52:10.453774 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:52:40.455543 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:52:40.455461 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:53:10.454531 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:53:10.454484 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:53:22.841653 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:22.841614 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jzwlg_dca34023-3797-4998-9674-ee14daf13450/global-pull-secret-syncer/0.log" Oct 30 18:53:23.180303 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:23.180278 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l6m5w_3b070856-f46c-4ebb-965c-d8aaacb18baf/connection-checker/0.log" Oct 30 18:53:23.303389 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:23.303349 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lfmlk_47a7c100-1107-4314-aa33-526e92b42c1f/connection-checker/0.log" Oct 30 18:53:23.431118 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:23.431034 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ztx4c_5084064f-9031-4a3c-b74f-46abf27982a3/connection-checker/0.log" Oct 30 18:53:23.726975 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:23.726886 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dw7mk_b440ae4e-411c-48fd-a8ed-45596e6b4147/konnectivity-agent/0.log" Oct 30 18:53:24.326676 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:24.326621 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-94.ec2.internal_00fcd38864cc26968aedfdf83ce5b096/haproxy/0.log" Oct 30 18:53:30.669069 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:30.669040 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-h8nms_28da65aa-8ccc-46c5-a8dc-06c341aa4e1c/kuadrant-console-plugin/0.log" Oct 30 18:53:32.749320 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.749285 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs"] Oct 30 18:53:32.749752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.749707 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a0fa4061-0c90-4c4c-bfc6-137110ca60ef" containerName="revoke-keys" Oct 30 18:53:32.749752 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.749720 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fa4061-0c90-4c4c-bfc6-137110ca60ef" containerName="revoke-keys" Oct 30 18:53:32.749825 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.749798 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fa4061-0c90-4c4c-bfc6-137110ca60ef" containerName="revoke-keys" Oct 30 18:53:32.753030 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.753007 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.775190 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.775158 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs"] Oct 30 18:53:32.854182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.854153 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-lib-modules\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.854182 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.854185 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-proc\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.854407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.854288 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-sys\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.854407 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.854340 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqqk\" (UniqueName: \"kubernetes.io/projected/8963cb3c-90d4-47e4-a96e-f94cfca713ad-kube-api-access-rdqqk\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.854480 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.854422 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-podres\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.864132 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.864112 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"openshift-service-ca.crt\"" Oct 30 18:53:32.864249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.864225 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jtmf9\"/\"default-dockercfg-6tdcw\"" Oct 30 18:53:32.864296 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.864286 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"kube-root-ca.crt\"" Oct 30 18:53:32.955077 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955047 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-sys\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955099 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqqk\" (UniqueName: \"kubernetes.io/projected/8963cb3c-90d4-47e4-a96e-f94cfca713ad-kube-api-access-rdqqk\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955131 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-podres\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955165 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-sys\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955177 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-lib-modules\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955245 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955217 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-proc\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955251 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-lib-modules\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955249 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-podres\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.955425 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.955302 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8963cb3c-90d4-47e4-a96e-f94cfca713ad-proc\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:32.973876 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:32.973839 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqqk\" (UniqueName: \"kubernetes.io/projected/8963cb3c-90d4-47e4-a96e-f94cfca713ad-kube-api-access-rdqqk\") pod \"perf-node-gather-daemonset-cx6xs\" (UID: \"8963cb3c-90d4-47e4-a96e-f94cfca713ad\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:33.065214 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:33.065143 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:33.150230 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:33.149510 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-r5zhx_9fe43c59-0934-4c18-84f1-381abdcf553d/cluster-monitoring-operator/0.log" Oct 30 18:53:33.381853 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:33.381797 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs"] Oct 30 18:53:33.384744 ip-10-0-143-94 kubenswrapper[5024]: W1030 18:53:33.384714 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8963cb3c_90d4_47e4_a96e_f94cfca713ad.slice/crio-65f9d0d7f18643847cbe3ee9cf3e7d2a360e9f94626d699bc681587bbc98bdad WatchSource:0}: Error finding container 65f9d0d7f18643847cbe3ee9cf3e7d2a360e9f94626d699bc681587bbc98bdad: Status 404 returned error can't find the container with id 65f9d0d7f18643847cbe3ee9cf3e7d2a360e9f94626d699bc681587bbc98bdad Oct 30 18:53:34.356787 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.356750 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" event={"ID":"8963cb3c-90d4-47e4-a96e-f94cfca713ad","Type":"ContainerStarted","Data":"0f8df301fc65605ecbb31c9aeb2b1798bf4b1dfaf23e40255b8aff8791938ca1"} Oct 30 18:53:34.357191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.356907 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:34.357191 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.356936 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" event={"ID":"8963cb3c-90d4-47e4-a96e-f94cfca713ad","Type":"ContainerStarted","Data":"65f9d0d7f18643847cbe3ee9cf3e7d2a360e9f94626d699bc681587bbc98bdad"} Oct 30 18:53:34.394330 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.394278 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" podStartSLOduration=2.394264846 podStartE2EDuration="2.394264846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:34.391838661 +0000 UTC m=+3165.212550437" watchObservedRunningTime="2026-10-30 18:53:34.394264846 +0000 UTC m=+3165.214976621" Oct 30 18:53:34.446161 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.446127 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvhb2_e3d39cba-f292-4a0e-99b8-12acac3cf477/node-exporter/0.log" Oct 30 18:53:34.549657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.549624 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvhb2_e3d39cba-f292-4a0e-99b8-12acac3cf477/kube-rbac-proxy/0.log" Oct 30 18:53:34.653612 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:34.653577 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvhb2_e3d39cba-f292-4a0e-99b8-12acac3cf477/init-textfile/0.log" Oct 30 18:53:35.849341 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:35.849311 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nm5jm_bc95cac7-936c-4a1f-b38a-5aa6a8fcf851/prometheus-operator/0.log" Oct 30 18:53:35.947508 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:35.947479 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-nm5jm_bc95cac7-936c-4a1f-b38a-5aa6a8fcf851/kube-rbac-proxy/0.log" Oct 30 18:53:36.052335 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:36.052305 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-59zkz_b70074d2-074f-4c37-99e8-841f060c98b8/prometheus-operator-admission-webhook/0.log" Oct 30 18:53:38.344906 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:38.344852 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-pllqc_16a01d63-08e8-4859-95c8-b89108043583/networking-console-plugin/0.log" Oct 30 18:53:39.284164 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:39.284132 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/1.log" Oct 30 18:53:39.292187 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:39.292156 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ts48c_d861baa6-45e0-44b6-9336-1ed846e00b49/console-operator/2.log" Oct 30 18:53:40.237565 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:40.237534 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bf7f4b65c-jmwkh_d27f19cf-63b5-497d-a0f5-27bcb1854f9d/console/0.log" Oct 30 18:53:40.373923 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:40.373895 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-cx6xs" Oct 30 18:53:40.402334 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:40.402307 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-j67kw_6eba5f4f-710d-4b2f-8efc-5c2a8e5b1035/download-server/0.log" Oct 30 18:53:40.455623 ip-10-0-143-94 kubenswrapper[5024]: E1030 18:53:40.455568 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bvmg" containerName="extractor" Oct 30 18:53:41.488657 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:41.488626 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ds2qd_62cbcc9b-a1ea-4304-9fe7-1805deaf6f03/volume-data-source-validator/0.log" Oct 30 18:53:43.988015 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:43.987975 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnrlr_3208b565-97ac-4ed9-9d36-a289544c5fd1/dns/0.log" Oct 30 18:53:44.101035 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:44.101006 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fnrlr_3208b565-97ac-4ed9-9d36-a289544c5fd1/kube-rbac-proxy/0.log" Oct 30 18:53:44.459936 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:44.459899 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f72ft_8cfe69b0-a077-4fd0-9f7e-a513c1b882bf/dns-node-resolver/0.log" Oct 30 18:53:45.762414 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:45.762370 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-644f4f5cdc-jgpvh_18db95bd-a7e5-49dc-9696-2e9b3fce9e4f/registry/0.log" Oct 30 18:53:46.138682 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:46.138582 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzdgf_7d5af711-aeb9-4984-b639-1d6b13db2909/node-ca/0.log" Oct 30 18:53:46.914838 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:46.914801 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-zsxmb_3552bb13-3271-4be5-8d76-5bac03ce89df/kube-auth-proxy/0.log" Oct 30 18:53:47.199688 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:47.199608 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hn66q_29abc014-6eb3-474b-95b4-cc26370fd325/istio-proxy/0.log" Oct 30 18:53:47.341862 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:47.341825 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n789k_8029cccf-1a8c-431e-a244-c840c94bb846/istio-proxy/0.log" Oct 30 18:53:47.467913 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:47.467831 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vftgt_9475a6a1-002e-49b3-bb80-67db243d4e0f/istio-proxy/0.log" Oct 30 18:53:47.901009 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:47.900975 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log" Oct 30 18:53:49.504259 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:49.504229 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-zsxmb_3552bb13-3271-4be5-8d76-5bac03ce89df/kube-auth-proxy/0.log" Oct 30 18:53:49.957185 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:49.957148 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hn66q_29abc014-6eb3-474b-95b4-cc26370fd325/istio-proxy/0.log" Oct 30 18:53:50.175995 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:50.175957 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n789k_8029cccf-1a8c-431e-a244-c840c94bb846/istio-proxy/0.log" Oct 30 18:53:50.391986 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:50.391961 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vftgt_9475a6a1-002e-49b3-bb80-67db243d4e0f/istio-proxy/0.log" Oct 30 18:53:51.077212 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:51.077181 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log" Oct 30 18:53:52.601122 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:52.601092 5024 scope.go:122] "RemoveContainer" containerID="34d01088c294bd2077b783076ebbe044ad6b557f4ea30f78c6d29f957e2c4d4b" Oct 30 18:53:52.651249 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:52.651220 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-zsxmb_3552bb13-3271-4be5-8d76-5bac03ce89df/kube-auth-proxy/0.log" Oct 30 18:53:52.924753 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:52.924724 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-hn66q_29abc014-6eb3-474b-95b4-cc26370fd325/istio-proxy/0.log" Oct 30 18:53:53.055705 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:53.055673 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n789k_8029cccf-1a8c-431e-a244-c840c94bb846/istio-proxy/0.log" Oct 30 18:53:53.188061 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:53.187985 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-vftgt_9475a6a1-002e-49b3-bb80-67db243d4e0f/istio-proxy/0.log" Oct 30 18:53:53.322271 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:53.322238 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dq8ks_47517669-cee7-4488-827e-b605f1cbfbc0/serve-healthcheck-canary/0.log" Oct 30 18:53:53.588659 ip-10-0-143-94 kubenswrapper[5024]: I1030 18:53:53.588558 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7765667bbf-4zpgd_b78ec231-1951-451f-8ec8-b70e20b98fc5/router/0.log"