Oct 30 22:00:39.073144 ip-10-0-134-125 systemd[1]: Starting Kubernetes Kubelet... Oct 30 22:00:39.458206 ip-10-0-134-125 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:39.458206 ip-10-0-134-125 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 22:00:39.458206 ip-10-0-134-125 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:39.458206 ip-10-0-134-125 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:39.462736 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462654 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.462736 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462736 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.462736 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462740 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462744 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462748 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462751 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462754 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462757 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462760 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462763 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462766 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462768 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462771 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462773 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462777 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462780 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462783 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462785 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462788 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462791 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462807 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462812 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.462844 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462817 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462821 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462824 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462827 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462830 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462832 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462835 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462837 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462840 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462843 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462846 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462849 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462852 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462855 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462858 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462861 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462863 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462866 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462868 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462871 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.463310 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462881 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462884 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462887 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462889 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462892 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462894 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462897 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462900 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462902 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462905 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462908 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462910 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462913 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462916 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462919 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462921 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462924 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462926 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462929 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462932 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.463775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462934 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462937 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462940 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462942 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462945 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462950 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462954 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462957 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462961 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462965 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462968 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462971 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462974 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462978 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462981 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462983 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462986 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462988 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462991 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.464261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462993 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462996 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.462999 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463001 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463004 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463007 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463009 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463012 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463015 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463018 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463020 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463023 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463026 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463028 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463032 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463035 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463038 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463040 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463043 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.464704 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463046 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463048 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463050 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463053 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463057 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.463059 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464259 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464267 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464271 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464274 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464277 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464279 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464282 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464285 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464287 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464290 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464293 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464295 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464298 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464300 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.465204 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464304 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464306 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464309 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464312 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464314 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464317 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464320 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464323 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464327 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464331 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464334 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464337 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464340 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464343 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464345 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464348 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464351 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464354 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464356 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.465692 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464359 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464361 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464364 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464367 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464370 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464372 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464375 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464377 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464380 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464383 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464385 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464388 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464390 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464393 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464396 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464399 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464401 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464404 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464407 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.466157 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464409 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464412 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464415 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464418 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464420 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464423 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464425 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464428 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464430 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464433 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464435 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464438 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464441 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464443 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464445 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464448 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464451 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464454 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464457 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464459 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.466622 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464462 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464465 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464468 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464470 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464473 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464476 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464478 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464481 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464483 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464486 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464489 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464491 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464494 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464496 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464499 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464502 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464504 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464507 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464509 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464512 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.467115 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464514 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464517 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464520 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464522 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464525 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464527 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464530 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464533 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464536 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464539 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464541 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464544 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464546 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.464551 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465090 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465097 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465100 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465103 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465106 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.467579 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465108 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465112 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465115 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465117 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465120 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465123 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465125 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465128 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465131 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465133 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465136 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465139 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465143 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465146 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465149 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465152 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465154 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465157 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465159 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.468040 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465162 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465164 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465169 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465173 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465175 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465178 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465181 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465183 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465186 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465189 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465191 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465194 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465196 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465199 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465201 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465204 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465207 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465209 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465212 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465214 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.468481 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465217 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465220 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465222 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465225 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465227 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465230 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465233 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465235 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465245 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465248 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465251 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465253 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465256 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465259 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465262 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465265 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465268 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465271 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465273 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.468970 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465276 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465279 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465282 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465285 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465288 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465290 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465293 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465296 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465298 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465302 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465306 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465310 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465313 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465316 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465318 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465321 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465323 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465325 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465328 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.469422 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465330 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465333 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465335 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465338 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465340 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465344 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465346 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465348 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465351 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465354 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465357 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465364 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465367 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465369 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465372 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465374 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465377 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465379 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465382 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465384 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.469914 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465387 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465389 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465391 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.465394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466236 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466248 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466255 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466260 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466265 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466268 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466272 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466276 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466280 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466283 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466286 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466290 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466293 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466296 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466299 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466302 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466305 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466308 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466333 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 22:00:39.470378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466337 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466341 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466344 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466348 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466356 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466359 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466363 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466366 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466369 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466372 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466375 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466378 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466381 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466386 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466389 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466393 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466395 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466398 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466402 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466406 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466410 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466413 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466416 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466419 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 22:00:39.470934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466424 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466427 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466430 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466434 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466437 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466440 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466443 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466446 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466449 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466454 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466458 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466462 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466465 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466468 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466472 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466475 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466478 2575 flags.go:64] FLAG: --help="false" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466481 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466485 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466488 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466491 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466494 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466498 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 22:00:39.471597 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466500 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466503 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466506 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466509 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466512 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466515 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466519 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466522 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466525 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466528 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466531 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466534 2575 flags.go:64] FLAG: --lock-file="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466537 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466540 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466544 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466550 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466553 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466556 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466561 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466564 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466568 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466571 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466574 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466578 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466582 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 22:00:39.472191 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466586 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466589 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466592 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466595 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466598 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466601 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466604 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466607 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466615 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466618 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466621 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466624 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466627 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466631 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466634 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466637 2575 flags.go:64] FLAG: --port="10250" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466640 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466644 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fed0b9b84326d2e7" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466647 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466650 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466653 2575 flags.go:64] FLAG: --register-node="true" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466658 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466662 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466665 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466668 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466672 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 22:00:39.473005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466676 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466679 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466683 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466686 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466689 2575 flags.go:64] FLAG: --runonce="false" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466692 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466695 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466698 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466701 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466704 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466707 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466711 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466715 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466717 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466721 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466725 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466730 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466734 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466739 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466743 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466752 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466756 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466760 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466766 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466770 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 22:00:39.474186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466776 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466781 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466787 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466792 2575 flags.go:64] FLAG: --v="2" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466812 2575 flags.go:64] FLAG: --version="false" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466820 2575 flags.go:64] FLAG: --vmodule="" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466828 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.466833 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467002 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467007 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467014 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467018 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467020 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467023 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467028 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467033 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467036 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467039 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467042 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467044 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.475238 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467047 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467050 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467053 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467055 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467058 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467061 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467064 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467066 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467069 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467071 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467074 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467077 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467079 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467082 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467086 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467089 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467091 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467094 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467098 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.476125 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467101 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467103 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467106 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467109 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467112 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467115 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467118 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467121 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467123 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467126 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467129 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467132 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467134 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467137 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467139 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467142 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467145 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467147 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467150 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467153 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.476986 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467155 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467158 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467161 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467163 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467166 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467168 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467171 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467175 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467177 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467180 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467184 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467188 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467191 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467194 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467197 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467199 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467203 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467206 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467209 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.477625 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467211 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467214 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467216 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467219 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467222 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467224 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467227 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467229 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467232 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467235 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467237 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467240 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467242 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467245 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467247 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467250 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467253 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467256 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467258 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467261 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.478470 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467264 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467267 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467270 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467272 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467277 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467279 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467282 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467285 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467288 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467290 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467293 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467297 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467299 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467302 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467305 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467307 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.479278 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.467334 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467370 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467374 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467377 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467380 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467382 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467385 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467388 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467390 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467393 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467396 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467399 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467401 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467404 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467406 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467409 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467413 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467416 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467418 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.479723 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467421 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467423 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467429 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467433 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467436 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467439 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467442 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467444 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467448 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467451 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467454 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467456 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467459 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467462 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467464 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467467 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467470 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467472 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467475 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.480298 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467477 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467480 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467482 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467485 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467487 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467490 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467492 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467495 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467498 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467500 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467504 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467508 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467510 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467513 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467515 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467519 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467522 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467524 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467527 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.480780 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467530 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467532 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467535 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467537 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467541 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467544 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467546 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467549 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467551 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467554 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467557 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467559 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467562 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467564 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467567 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467570 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467572 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467575 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467577 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467581 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.481284 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467584 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467588 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467592 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467596 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467599 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467602 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467604 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467607 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467610 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467613 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467615 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467618 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467620 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467623 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467625 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467628 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467630 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467633 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467636 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467638 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.482013 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467641 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467643 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467645 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467648 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467651 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467654 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467656 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467659 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467661 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.467664 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.475122 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.475141 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.475163 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.475171 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475257 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475266 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475272 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475278 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475283 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475287 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.482515 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475291 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475296 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475301 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475305 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475310 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475319 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475323 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475328 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475333 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475337 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475342 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475346 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475350 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475354 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475359 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475364 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475369 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475373 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475377 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.483035 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475382 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475386 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475391 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475396 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475400 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475405 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475409 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475414 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475418 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475422 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475427 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475431 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475435 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475439 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475444 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475448 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475453 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475457 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475461 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.483504 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475466 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475470 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475475 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475480 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475488 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475493 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475497 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475502 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475506 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475511 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475515 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475519 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475523 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475528 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475532 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475537 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475541 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475545 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475550 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.484103 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475554 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475558 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475563 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475567 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475577 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475581 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475586 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475590 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475595 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475599 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475603 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475607 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475612 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475616 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475621 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475625 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475630 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475634 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.484776 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475638 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475643 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475647 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475651 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475656 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475660 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475665 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475669 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475673 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475679 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475685 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475690 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475694 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475698 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475702 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475706 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475711 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475715 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475719 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475723 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.485420 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475727 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.475734 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.475777 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476053 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476062 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476069 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476073 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476078 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476083 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476087 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476092 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476097 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476101 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476106 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476111 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.486221 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476115 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476120 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476124 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476128 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476132 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476136 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476140 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476144 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476148 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476153 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476157 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476161 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476166 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476170 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476174 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476178 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476183 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476187 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476191 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476195 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.486860 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476199 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476203 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476208 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476213 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476218 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476225 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476230 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476234 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476240 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476245 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476250 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476254 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476258 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476263 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476267 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476271 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476275 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476279 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476283 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.487700 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476287 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476291 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476296 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476300 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476304 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476309 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476313 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476317 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476322 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476326 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476330 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476334 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476340 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476345 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476349 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476353 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476358 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476362 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476366 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476370 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.488246 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476374 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476381 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476388 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476392 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476397 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476402 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476406 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476410 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476414 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476418 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476423 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476427 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476431 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476436 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476440 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476444 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476449 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476453 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476457 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476461 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.488726 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476465 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476469 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476474 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476478 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476482 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476487 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476491 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476495 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476499 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476503 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476507 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476511 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476516 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476520 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476524 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.489247 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.476565 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476619 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476627 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476632 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476637 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476641 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476646 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476650 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476655 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476659 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476663 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476667 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476671 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476675 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476680 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476684 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476688 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476692 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476696 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:39.489609 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476700 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476705 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476709 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476714 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476718 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476723 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476727 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476731 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476735 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476739 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476744 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476748 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476752 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476756 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476761 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476765 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476770 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476774 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476778 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:39.490092 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476783 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476787 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476791 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476810 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476815 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476819 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476823 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476828 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476832 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476836 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476840 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476845 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476849 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476856 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476861 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476865 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476870 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476874 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476878 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476882 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:39.490566 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476886 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476890 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476894 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476902 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476908 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476913 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476917 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476921 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476927 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476931 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476935 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476939 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476943 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476948 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476952 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476956 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476960 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476965 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476969 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:39.491105 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476973 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476977 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476981 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476985 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476989 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.476994 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477002 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477006 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477011 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477016 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477020 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477025 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477030 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477034 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477039 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477043 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477047 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477051 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477055 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477061 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:39.491695 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477067 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477072 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477077 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477081 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477086 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477090 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477094 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477098 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477105 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:39.477109 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.477191 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.479726 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.480753 2575 server.go:1014] "Starting client certificate rotation" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.480829 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.480873 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.486064 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:00:39.492182 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.487809 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:00:39.499533 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.499514 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 22:00:39.502626 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.502611 2575 log.go:25] "Validated CRI v1 image API" Oct 30 22:00:39.503361 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.503332 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 22:00:39.507262 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.507244 2575 fs.go:135] Filesystem UUIDs: map[677135cd-0124-43ba-b127-2dc3a14f6b76:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 803169c3-f9d2-4c18-aeb3-b0685de0b202:/dev/nvme0n1p3] Oct 30 22:00:39.507318 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.507262 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 22:00:39.512681 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.512568 2575 manager.go:217] Machine: {Timestamp:2026-10-30 22:00:39.511222384 +0000 UTC m=+0.321314505 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103561 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e72905c7e9bdf5d2b66e83512936 SystemUUID:ec27e729-05c7-e9bd-f5d2-b66e83512936 BootID:3683f71e-0ee9-4424-8a12-d1eae07ac313 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fe:bb:7a:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fe:bb:7a:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:d0:63:9c:f5:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 22:00:39.512681 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.512680 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 22:00:39.512857 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.512805 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 22:00:39.515644 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515621 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 22:00:39.515787 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515646 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 22:00:39.515849 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515788 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 22:00:39.515849 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515812 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 22:00:39.515849 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515845 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:00:39.515925 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.515858 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:00:39.516508 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.516497 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 22:00:39.518980 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.518968 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 22:00:39.519101 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519092 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 22:00:39.519751 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519740 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 22:00:39.519809 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519758 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 22:00:39.519809 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519771 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 22:00:39.519809 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519780 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 22:00:39.519809 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.519787 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 22:00:39.520382 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.520369 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:00:39.522677 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.522649 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 22:00:39.528576 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.528557 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:00:39.529305 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529293 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 22:00:39.529349 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529323 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 22:00:39.529477 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529467 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 22:00:39.529527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529483 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 22:00:39.529527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529490 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 22:00:39.529527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529496 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 22:00:39.529527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529504 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 22:00:39.529527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.529509 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 22:00:39.530086 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530078 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 22:00:39.530115 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530087 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 22:00:39.530115 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530094 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 22:00:39.530115 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530100 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 22:00:39.530115 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530108 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 22:00:39.530115 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530117 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 22:00:39.530862 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530854 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 22:00:39.530892 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.530863 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 22:00:39.534610 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.534599 2575 server.go:1263] "Started kubelet" Oct 30 22:00:39.534730 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.534697 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 22:00:39.534840 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.534688 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 22:00:39.534840 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.534807 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 22:00:39.535423 ip-10-0-134-125 systemd[1]: Started Kubernetes Kubelet. Oct 30 22:00:39.536033 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.536018 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 22:00:39.538782 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.538768 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 22:00:39.540620 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.540591 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 22:00:39.542286 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.542266 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 22:00:39.543160 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543143 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 22:00:39.543303 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.543282 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:39.543769 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543750 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 22:00:39.543862 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543753 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 22:00:39.543862 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543812 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 22:00:39.543974 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543926 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 22:00:39.543974 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.543937 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 22:00:39.544077 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544061 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 22:00:39.544077 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544072 2575 factory.go:55] Registering systemd factory Oct 30 22:00:39.544077 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544078 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 22:00:39.544278 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544262 2575 factory.go:153] Registering CRI-O factory Oct 30 22:00:39.544335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544279 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 22:00:39.544335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544294 2575 factory.go:103] Registering Raw factory Oct 30 22:00:39.544335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.544303 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 22:00:39.545091 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.545077 2575 manager.go:319] Starting recovery of all containers Oct 30 22:00:39.557326 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.557299 2575 manager.go:324] Recovery completed Oct 30 22:00:39.560729 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.560701 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 22:00:39.563606 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.563593 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.565566 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.565551 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.565620 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.565582 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.565620 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.565597 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.566152 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566137 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 22:00:39.566152 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566150 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 22:00:39.566293 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566169 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 22:00:39.566293 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566181 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.566937 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566921 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.567005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566951 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.567005 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.566962 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.567365 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.567356 2575 policy_none.go:50] "Start" Oct 30 22:00:39.567400 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.567370 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 22:00:39.567400 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.567379 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 22:00:39.568655 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.568639 2575 policy_none.go:44] "Start" Oct 30 22:00:39.606351 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.606320 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 22:00:39.611030 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611017 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 22:00:39.611130 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.611084 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 22:00:39.611130 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611096 2575 server.go:87] "Starting device plugin registration server" Oct 30 22:00:39.611327 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611314 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 22:00:39.611372 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611329 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 22:00:39.611449 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611433 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 22:00:39.611551 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611537 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 22:00:39.611551 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.611550 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 22:00:39.612559 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.612150 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 22:00:39.612559 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.612193 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:39.668642 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.668613 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 22:00:39.669965 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.669943 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 22:00:39.669965 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.669966 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 22:00:39.670106 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.669988 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 22:00:39.670106 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.670020 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 22:00:39.671534 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.671511 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-125.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 22:00:39.672311 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.671453 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-2.compute.internal.18e36da6bd7b6fa0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:39.534571424 +0000 UTC m=+0.344663546,LastTimestamp:2026-10-30 22:00:39.534571424 +0000 UTC m=+0.344663546,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-2.compute.internal,}" Oct 30 22:00:39.711698 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.711628 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.712683 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.712663 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.712767 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.712700 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.712767 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.712714 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.712767 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.712739 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.731155 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.731074 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-2.compute.internal.18e36da6bf54646e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-125.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:39.565567086 +0000 UTC m=+0.375659212,LastTimestamp:2026-10-30 22:00:39.565567086 +0000 UTC m=+0.375659212,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-2.compute.internal,}" Oct 30 22:00:39.731297 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.731126 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 22:00:39.731366 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.731331 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-125.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 22:00:39.731366 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.731350 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 22:00:39.771495 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.771457 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal"] Oct 30 22:00:39.771582 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.771532 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.773479 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.773460 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 22:00:39.773539 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.773525 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bscjc" Oct 30 22:00:39.777330 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.777313 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.777422 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.777340 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.777422 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.777349 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.778379 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.778367 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.778526 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.778511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.778575 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.778542 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.779123 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779109 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.779187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779129 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.779187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779139 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.779187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779168 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.779275 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779192 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.779275 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.779202 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.780164 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.780150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.780215 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.780182 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:39.780780 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.780753 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-125.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 22:00:39.781294 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.781277 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:39.781344 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.781301 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:39.781344 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.781311 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:39.814270 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.814243 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.818171 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.818156 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.822566 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.822498 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-2.compute.internal.18e36da6bf54bc2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-125.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:39.565589546 +0000 UTC m=+0.375681667,LastTimestamp:2026-10-30 22:00:39.565589546 +0000 UTC m=+0.375681667,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-2.compute.internal,}" Oct 30 22:00:39.822680 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.822662 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 22:00:39.846280 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.846256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.846390 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.846286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.846390 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.846310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.882408 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:39.882383 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947063 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947063 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947264 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947264 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947109 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947264 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:39.947264 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:39.947138 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.029767 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.029532 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.083332 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.083292 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:40.085739 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.085718 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:40.085878 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.085751 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:40.085878 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.085762 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:40.085878 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.085789 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.117209 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.117187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.120870 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.120719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.140685 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.140663 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:40.140769 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.140688 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-125.us-west-2.compute.internal\": node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.250382 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.250352 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.351213 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.351189 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.451434 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.451392 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.480195 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.480156 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 22:00:40.480676 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.480326 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a56facd814ea6491ea434b7fb55cee90-852268fd564bd9ac.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-125.us-west-2.compute.internal.18e36da6bf54646e\": read tcp 10.0.134.125:58594->3.150.186.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-125.us-west-2.compute.internal.18e36da6bf54646e default 5264 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-125.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:39 +0000 UTC,LastTimestamp:2026-10-30 22:00:39.777326882 +0000 UTC m=+0.587419003,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.us-west-2.compute.internal,}" Oct 30 22:00:40.552598 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.552566 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.653559 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.653480 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.754204 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.754165 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.776960 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.776929 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:40 +0000 UTC" deadline="2028-08-10 21:32:38.191141674 +0000 UTC" Oct 30 22:00:40.776960 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.776958 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15599h31m57.414188475s" Oct 30 22:00:40.780033 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.780013 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:40.837023 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.836997 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:40.838073 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.838054 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:40.838189 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.838090 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:40.838189 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.838105 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:40.854716 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.854693 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:40.896158 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:40.896139 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:40.955344 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:40.955285 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:41.055976 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.055940 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:00:41.066171 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.066151 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:41.067992 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.067975 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:41.144043 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.144016 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:41.212884 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.212812 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:00:41.214089 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.214071 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:00:41.254336 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.254300 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52151281fe9c0d516ce7c27b4aad2f39.slice/crio-efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8 WatchSource:0}: Error finding container efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8: Status 404 returned error can't find the container with id efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8 Oct 30 22:00:41.259727 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.259700 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:00:41.262533 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.262513 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234561db2c60215956651b28b401629e.slice/crio-922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82 WatchSource:0}: Error finding container 922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82: Status 404 returned error can't find the container with id 922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82 Oct 30 22:00:41.282425 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.282404 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:00:41.522512 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.522429 2575 apiserver.go:52] "Watching apiserver" Oct 30 22:00:41.544528 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.544497 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 22:00:41.639819 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.639791 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 22:00:41.640150 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.640132 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-grfcj","openshift-network-operator/iptables-alerter-rt97s","kube-system/konnectivity-agent-5r47b","openshift-cluster-node-tuning-operator/tuned-zjfbp","openshift-multus/multus-additional-cni-plugins-n6z8w","openshift-ovn-kubernetes/ovnkube-node-857zd","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal","openshift-multus/multus-xwcxh","openshift-multus/network-metrics-daemon-m4j6p","kube-system/global-pull-secret-syncer-qhvzk","openshift-dns/node-resolver-9hlwl","openshift-image-registry/node-ca-rnwfr"] Oct 30 22:00:41.642212 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.642192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.643368 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.643344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.644609 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.644592 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.645666 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.645647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.646748 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.646726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.647963 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.647949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.649395 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.649081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.650512 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.650494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.652010 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.651990 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.653865 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.653847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:41.653979 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.653929 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:41.655201 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.655181 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.655272 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.655253 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:41.656541 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.656602 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.656571 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:41.656602 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656674 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656674 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.656738 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.656738 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656799 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656799 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.656928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.656928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.656928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ht9\" (UniqueName: \"kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.656997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gcx\" (UniqueName: \"kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clhm\" (UniqueName: \"kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657427 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.657702 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.657589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.674697 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.674653 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerStarted","Data":"922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82"} Oct 30 22:00:41.675928 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.675900 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"52151281fe9c0d516ce7c27b4aad2f39","Type":"ContainerStarted","Data":"efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8"} Oct 30 22:00:41.702919 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.702895 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\"" Oct 30 22:00:41.703037 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.702970 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.703037 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.702979 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\"" Oct 30 22:00:41.703037 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.702966 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.703037 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.702970 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.703037 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703012 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 22:00:41.703230 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703041 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.703230 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703015 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 22:00:41.703805 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703789 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 22:00:41.703890 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703863 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\"" Oct 30 22:00:41.703890 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.703880 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 22:00:41.704591 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.704575 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.704747 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.704734 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.704985 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.704969 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\"" Oct 30 22:00:41.747625 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.747592 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 22:00:41.755027 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.755005 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\"" Oct 30 22:00:41.758261 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.758310 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.758310 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clhm\" (UniqueName: \"kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.758413 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758438 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758560 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758649 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.758692 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758725 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758784 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k295f\" (UniqueName: \"kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc72d\" (UniqueName: \"kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.758950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759002 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.759143 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759215 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759275 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759361 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759389 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.759659 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hm6\" (UniqueName: \"kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xqcb\" (UniqueName: \"kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hjd\" (UniqueName: \"kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.760259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759830 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dpd\" (UniqueName: \"kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759936 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.759987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760141 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.761016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760163 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ht9\" (UniqueName: \"kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.761612 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.760975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.761026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gcx\" (UniqueName: \"kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.761096 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762187 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.761284 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.762640 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.762597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762640 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.762624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.762834 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.762809 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.763053 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.763036 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 22:00:41.763110 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.763078 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7kbs" Oct 30 22:00:41.811891 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.811791 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 22:00:41.811891 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.811826 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.811891 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.811791 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.812151 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.812024 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.812151 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.812099 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.812346 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.812325 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 22:00:41.812414 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.812371 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.819774 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.819753 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\"" Oct 30 22:00:41.820057 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820041 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 22:00:41.820114 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820073 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 22:00:41.820114 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820087 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 22:00:41.820114 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820091 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 22:00:41.820114 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820101 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820127 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820178 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820199 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820211 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820224 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 22:00:41.820284 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820244 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 22:00:41.820457 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820293 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\"" Oct 30 22:00:41.820457 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.820348 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\"" Oct 30 22:00:41.827760 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.827740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clhm\" (UniqueName: \"kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.829232 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.829212 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ht9\" (UniqueName: \"kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.830497 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.830479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gcx\" (UniqueName: \"kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.861757 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.861851 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.861851 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.861851 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861914 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861922 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hm6\" (UniqueName: \"kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.861981 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.861959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqcb\" (UniqueName: \"kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.862196 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hjd\" (UniqueName: \"kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.862196 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.862303 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dpd\" (UniqueName: \"kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.862380 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.862423 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.862464 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.862512 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.862563 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.862563 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.862666 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.862719 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862648 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.862719 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862704 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.862797 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.862797 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.862873 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.862821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.863027 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863064 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863064 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.863157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:41.863157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.863157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863168 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.863236 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863274 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863335 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863327 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863377 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.863408 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:42.363390726 +0000 UTC m=+3.173482861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863690 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.863842 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863693 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863756 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863918 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k295f\" (UniqueName: \"kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863952 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863981 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.863970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc72d\" (UniqueName: \"kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864068 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864132 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864206 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.864696 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864231 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864240 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.864310 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864271 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.864367 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:42.364352927 +0000 UTC m=+3.174445042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864337 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.865565 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.864476 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.866240 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.866223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.931098 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.930905 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:41.931098 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.930934 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:41.931098 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.930950 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:41.931098 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:41.931026 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:42.431004471 +0000 UTC m=+3.241096608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:41.932654 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.932618 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hm6\" (UniqueName: \"kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:41.933318 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.933290 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqcb\" (UniqueName: \"kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.933760 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.933727 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hjd\" (UniqueName: \"kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:41.933864 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.933733 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dpd\" (UniqueName: \"kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:41.933864 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.933791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc72d\" (UniqueName: \"kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:41.933941 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.933893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k295f\" (UniqueName: \"kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.954202 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.954177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xwcxh" Oct 30 22:00:41.960027 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.960008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:00:41.963261 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.963240 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b562927_1ada_4f1e_8801_a0ef71f9e654.slice/crio-c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988 WatchSource:0}: Error finding container c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988: Status 404 returned error can't find the container with id c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988 Oct 30 22:00:41.966982 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.966961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:41.968629 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.968604 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ef4db2_e1f8_4784_ad09_f8c612620c64.slice/crio-eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783 WatchSource:0}: Error finding container eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783: Status 404 returned error can't find the container with id eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783 Oct 30 22:00:41.974684 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.974661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:00:41.975775 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.975752 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f1f8ea_fb7b_4bfb_be7b_cfe42a2ea4f3.slice/crio-35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0 WatchSource:0}: Error finding container 35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0: Status 404 returned error can't find the container with id 35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0 Oct 30 22:00:41.982448 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.982427 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5fbe95_8587_45cc_afab_de9dc77674ba.slice/crio-8c80c1a0569274ae44817514966e557a909029e26ed7a7365711cf77cd8dae56 WatchSource:0}: Error finding container 8c80c1a0569274ae44817514966e557a909029e26ed7a7365711cf77cd8dae56: Status 404 returned error can't find the container with id 8c80c1a0569274ae44817514966e557a909029e26ed7a7365711cf77cd8dae56 Oct 30 22:00:41.987883 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.987855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:00:41.994542 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.994525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:00:41.995480 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:41.995458 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21847fd_7c43_4822_bc04_a64dba1fc77c.slice/crio-62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff WatchSource:0}: Error finding container 62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff: Status 404 returned error can't find the container with id 62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff Oct 30 22:00:41.996519 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:41.996505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:00:42.001313 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.001288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:00:42.003006 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:42.002974 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339dc820_f292_4db9_9ad8_206840a9c6b4.slice/crio-4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519 WatchSource:0}: Error finding container 4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519: Status 404 returned error can't find the container with id 4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519 Oct 30 22:00:42.003654 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.003639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:00:42.006651 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:42.006629 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d296dcb_2c18_45b4_9ae4_97fc339d9095.slice/crio-0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec WatchSource:0}: Error finding container 0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec: Status 404 returned error can't find the container with id 0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec Oct 30 22:00:42.011603 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:42.011584 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a903ca_0e8b_4fc1_a762_85bf02c32040.slice/crio-5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f WatchSource:0}: Error finding container 5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f: Status 404 returned error can't find the container with id 5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f Oct 30 22:00:42.013434 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:00:42.013415 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c511576_fd22_4f42_8c20_76c0f1d21464.slice/crio-5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a WatchSource:0}: Error finding container 5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a: Status 404 returned error can't find the container with id 5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a Oct 30 22:00:42.368852 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.368802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:42.369016 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.368872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:42.369016 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.368975 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:42.369016 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.368976 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:42.369116 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.369034 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.369017387 +0000 UTC m=+4.179109516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:42.369116 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.369047 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.369041533 +0000 UTC m=+4.179133643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:42.469815 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.469782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:42.470012 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.469964 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:42.470012 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.469980 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:42.470012 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.469994 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:42.470193 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:42.470057 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:43.470038101 +0000 UTC m=+4.280130214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:42.680519 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.680423 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" event={"ID":"a2a903ca-0e8b-4fc1-a762-85bf02c32040","Type":"ContainerStarted","Data":"5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f"} Oct 30 22:00:42.681806 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.681715 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec"} Oct 30 22:00:42.683405 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.683378 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerStarted","Data":"62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff"} Oct 30 22:00:42.685093 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.685065 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5r47b" event={"ID":"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3","Type":"ContainerStarted","Data":"35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0"} Oct 30 22:00:42.686491 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.686445 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9hlwl" event={"ID":"64ef4db2-e1f8-4784-ad09-f8c612620c64","Type":"ContainerStarted","Data":"eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783"} Oct 30 22:00:42.687934 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.687910 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xwcxh" event={"ID":"5b562927-1ada-4f1e-8801-a0ef71f9e654","Type":"ContainerStarted","Data":"c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988"} Oct 30 22:00:42.689128 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.689102 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rt97s" event={"ID":"4c511576-fd22-4f42-8c20-76c0f1d21464","Type":"ContainerStarted","Data":"5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a"} Oct 30 22:00:42.690369 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.690341 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnwfr" event={"ID":"339dc820-f292-4db9-9ad8-206840a9c6b4","Type":"ContainerStarted","Data":"4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519"} Oct 30 22:00:42.691658 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.691629 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" event={"ID":"3f5fbe95-8587-45cc-afab-de9dc77674ba","Type":"ContainerStarted","Data":"8c80c1a0569274ae44817514966e557a909029e26ed7a7365711cf77cd8dae56"} Oct 30 22:00:42.765916 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.765870 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:42 +0000 UTC" deadline="2028-04-20 01:49:43.635636327 +0000 UTC" Oct 30 22:00:42.765916 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:42.765909 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12891h49m0.869731725s" Oct 30 22:00:43.377769 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.377723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:43.377976 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.377846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:43.378134 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.378113 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:43.378221 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.378207 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.378183451 +0000 UTC m=+6.188275580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:43.378310 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.378293 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:43.378360 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.378332 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.378319266 +0000 UTC m=+6.188411392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:43.478957 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.478913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:43.479154 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.479121 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:43.479154 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.479151 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:43.479281 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.479165 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:43.479281 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.479265 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.479243824 +0000 UTC m=+6.289335953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:43.673766 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.673689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:43.673955 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.673832 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:43.673955 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.673939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:43.674080 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.674037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:43.674134 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.674098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:43.674181 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:43.674168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:43.766694 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.766588 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:42 +0000 UTC" deadline="2028-07-21 07:10:11.476431834 +0000 UTC" Oct 30 22:00:43.767259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:43.766689 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15105h9m27.709745201s" Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.398051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.398123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.398266 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.398332 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:49.398311629 +0000 UTC m=+10.208403762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.398408 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:45.398475 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.398444 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:49.398431489 +0000 UTC m=+10.208523616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:45.499481 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.499445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:45.499650 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.499633 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:45.499746 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.499712 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:45.499746 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.499729 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:45.499887 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.499800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:49.499778183 +0000 UTC m=+10.309870315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.671899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.671922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.672030 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.672133 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.672171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:45.672316 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:45.672239 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:45.713647 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.713020 2575 generic.go:356] "Generic (PLEG): container finished" podID="234561db2c60215956651b28b401629e" containerID="963777689b82e6bfc9ac84800220ad1a55e5ab30342f1be017065199efde3df1" exitCode=0 Oct 30 22:00:45.713647 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.713595 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerDied","Data":"963777689b82e6bfc9ac84800220ad1a55e5ab30342f1be017065199efde3df1"} Oct 30 22:00:45.719157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:45.719084 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"52151281fe9c0d516ce7c27b4aad2f39","Type":"ContainerStarted","Data":"03a494f075f35f2e0994bb9cc02e7d52c68a29bfe0922ebd6d79da256f320468"} Oct 30 22:00:46.729977 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:46.729269 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerStarted","Data":"e0f8ebd0c24eab02ff2b40772df168f5db519aacc53466029c47633531b328f2"} Oct 30 22:00:46.733879 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:46.733349 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="8b40734068026c85e66c70dc9b9330c890b6412a5b786a249228b3372a668757" exitCode=0 Oct 30 22:00:46.734082 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:46.733868 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"8b40734068026c85e66c70dc9b9330c890b6412a5b786a249228b3372a668757"} Oct 30 22:00:46.850681 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:46.850520 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" podStartSLOduration=5.850502055 podStartE2EDuration="5.850502055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:45.966059762 +0000 UTC m=+6.776151894" watchObservedRunningTime="2026-10-30 22:00:46.850502055 +0000 UTC m=+7.660594190" Oct 30 22:00:46.850996 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:46.850714 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" podStartSLOduration=5.850700593 podStartE2EDuration="5.850700593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:46.850056126 +0000 UTC m=+7.660148258" watchObservedRunningTime="2026-10-30 22:00:46.850700593 +0000 UTC m=+7.660792725" Oct 30 22:00:47.672067 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:47.672036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:47.672246 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:47.672146 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:47.672246 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:47.672221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:47.672339 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:47.672307 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:47.672381 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:47.672367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:47.672440 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:47.672425 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:47.743847 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:47.743750 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5r47b" event={"ID":"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3","Type":"ContainerStarted","Data":"12addaf2acec9212c2fa02e6987305ffa6078476c50ee71f5b45df644e3ce3d9"} Oct 30 22:00:47.863244 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:47.863185 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5r47b" podStartSLOduration=3.27707895 podStartE2EDuration="7.863167245s" totalImagesPullingTime="4.586088295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:47.863021903 +0000 UTC m=+8.673114035" watchObservedRunningTime="2026-10-30 22:00:47.863167245 +0000 UTC m=+8.673259380" Oct 30 22:00:48.748337 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:48.748305 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rnwfr" event={"ID":"339dc820-f292-4db9-9ad8-206840a9c6b4","Type":"ContainerStarted","Data":"1730031005e12715254a4f3caebb03e86afe69fa679839c404d501911ea96ba7"} Oct 30 22:00:49.066947 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.066866 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:49.067678 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.067652 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:49.192621 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.192570 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rnwfr" podStartSLOduration=3.4268541790000002 podStartE2EDuration="9.192550785s" totalImagesPullingTime="5.765696606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:48.869171005 +0000 UTC m=+9.679263139" watchObservedRunningTime="2026-10-30 22:00:49.192550785 +0000 UTC m=+10.002642917" Oct 30 22:00:49.437697 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.437619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:49.437697 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.437692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:49.437926 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.437837 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:49.437926 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.437897 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:57.437880174 +0000 UTC m=+18.247972286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:49.438049 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.437964 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:49.438049 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.437998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:00:57.437987279 +0000 UTC m=+18.248079390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:49.538377 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.538325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:49.538548 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.538502 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:49.538548 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.538524 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:49.538548 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.538535 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:49.538716 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.538588 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:57.538571152 +0000 UTC m=+18.348663281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.671154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.671264 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.671326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.671440 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.671498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:49.672432 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:49.671565 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:49.752599 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:49.752530 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" event={"ID":"a2a903ca-0e8b-4fc1-a762-85bf02c32040","Type":"ContainerStarted","Data":"32adc4bc389568a4d97991c56ec330b508d0080410e40af39e052806f209f7c5"} Oct 30 22:00:50.756956 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:50.756875 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9hlwl" event={"ID":"64ef4db2-e1f8-4784-ad09-f8c612620c64","Type":"ContainerStarted","Data":"7a255b7a15992be5b3cabd0e4669958146eadc8273cb3c16334d7888b9d3786d"} Oct 30 22:00:50.759634 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:50.759612 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:00:50.760355 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:50.760336 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xwcxh" event={"ID":"5b562927-1ada-4f1e-8801-a0ef71f9e654","Type":"ContainerStarted","Data":"ea72aef366bd3e11542bc0bc485bc7b00bd055b26bea7f9f470fefa863ceac6a"} Oct 30 22:00:50.879426 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:50.879374 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9hlwl" podStartSLOduration=2.397632656 podStartE2EDuration="10.8793552s" totalImagesPullingTime="8.481722544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:50.879041689 +0000 UTC m=+11.689133821" watchObservedRunningTime="2026-10-30 22:00:50.8793552 +0000 UTC m=+11.689447332" Oct 30 22:00:51.016205 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.016163 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 22:00:51.619510 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.619197 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:00:51.016180124Z","UUID":"ba5a156e-4fd2-40b6-8281-f3df958a2ef2","Handler":null,"Name":"","Endpoint":""} Oct 30 22:00:51.621025 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.621006 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 22:00:51.621025 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.621029 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.674504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:51.674683 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.674784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:51.674910 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.674978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:51.675485 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:51.675077 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:51.764588 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.763953 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="a56d4310d9826bb2a1e2d028fc95487e4f866b66320bc08bde846d4ab8c3304b" exitCode=0 Oct 30 22:00:51.764588 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.763997 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"a56d4310d9826bb2a1e2d028fc95487e4f866b66320bc08bde846d4ab8c3304b"} Oct 30 22:00:51.766515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.766447 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" event={"ID":"a2a903ca-0e8b-4fc1-a762-85bf02c32040","Type":"ContainerStarted","Data":"1bdc1cd3358aa49788d9c5aa09ea0b378b0a7bee6162f74a4303d297f97c4b76"} Oct 30 22:00:51.892836 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:51.892717 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xwcxh" podStartSLOduration=3.41874801 podStartE2EDuration="11.89269889s" totalImagesPullingTime="8.47395088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:51.009074761 +0000 UTC m=+11.819166916" watchObservedRunningTime="2026-10-30 22:00:51.89269889 +0000 UTC m=+12.702791022" Oct 30 22:00:52.771076 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:52.771042 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" event={"ID":"a2a903ca-0e8b-4fc1-a762-85bf02c32040","Type":"ContainerStarted","Data":"c2c606f93cb9f7480de76c931a98f28637af2cbd318a77ca1914457450168790"} Oct 30 22:00:52.772814 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:52.772772 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rt97s" event={"ID":"4c511576-fd22-4f42-8c20-76c0f1d21464","Type":"ContainerStarted","Data":"acccf67e337f480a3bb7088627d1760937020f26722e8dc1ceb8c0067b043ffb"} Oct 30 22:00:52.895731 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:52.895682 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" podStartSLOduration=2.690381114 podStartE2EDuration="12.895668471s" totalImagesPullingTime="10.205287357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:52.89473509 +0000 UTC m=+13.704827223" watchObservedRunningTime="2026-10-30 22:00:52.895668471 +0000 UTC m=+13.705760620" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.671262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.671284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.671273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:53.671422 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:53.671458 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:53.671535 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:53.671512 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:53.777515 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.777481 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="d9471c9d9858ad40b06fc72d93852c4116529bf0a933b7338085d7f5a9e108b2" exitCode=0 Oct 30 22:00:53.778097 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.777564 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"d9471c9d9858ad40b06fc72d93852c4116529bf0a933b7338085d7f5a9e108b2"} Oct 30 22:00:53.779363 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.779333 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" event={"ID":"3f5fbe95-8587-45cc-afab-de9dc77674ba","Type":"ContainerStarted","Data":"1866063da6a4c770be49579702ffa6d0d40ac2341e5f5018bacebe230dfea786"} Oct 30 22:00:53.906939 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.906893 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rt97s" podStartSLOduration=4.874240896 podStartE2EDuration="13.906879536s" totalImagesPullingTime="9.03263864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:53.024031365 +0000 UTC m=+13.834123508" watchObservedRunningTime="2026-10-30 22:00:53.906879536 +0000 UTC m=+14.716971667" Oct 30 22:00:53.938007 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.937924 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:53.938148 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.938068 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:00:53.938751 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:53.938726 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:00:54.030768 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:54.030702 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" podStartSLOduration=3.193253755 podStartE2EDuration="14.030682125s" totalImagesPullingTime="10.83742837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:54.030184145 +0000 UTC m=+14.840276279" watchObservedRunningTime="2026-10-30 22:00:54.030682125 +0000 UTC m=+14.840774256" Oct 30 22:00:55.671059 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:55.670811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:55.671627 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:55.670818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:55.671627 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:55.671183 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:55.671627 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:55.670866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:55.671627 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:55.671208 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:55.671627 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:55.671286 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:55.786157 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:55.786122 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="f1d3aac38aba77f84168a79058d6c0ec85a796f7cdf0f2cb5460db26763654d8" exitCode=0 Oct 30 22:00:55.786303 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:55.786179 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"f1d3aac38aba77f84168a79058d6c0ec85a796f7cdf0f2cb5460db26763654d8"} Oct 30 22:00:57.497593 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.497289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:57.498039 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.497592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:57.498039 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.497461 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:57.498039 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.497700 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:57.498039 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.497709 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.497692694 +0000 UTC m=+34.307784808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:57.498039 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.497752 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret podName:e4971693-7a0b-46a6-80bc-ad5329f213a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.497735775 +0000 UTC m=+34.307827886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret") pod "global-pull-secret-syncer-qhvzk" (UID: "e4971693-7a0b-46a6-80bc-ad5329f213a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:57.597898 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.597860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:57.598062 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.598042 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:57.598127 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.598064 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:57.598127 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.598079 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zvbx for pod openshift-network-diagnostics/network-check-target-grfcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:57.598215 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.598136 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx podName:91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:13.598122605 +0000 UTC m=+34.408214737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zvbx" (UniqueName: "kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx") pod "network-check-target-grfcj" (UID: "91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:57.671203 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.671173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:57.671357 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.671306 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:57.671436 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.671390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:57.671532 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.671514 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:57.671601 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.671547 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:57.671650 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:57.671629 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:57.793915 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.793836 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:00:57.794259 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794227 2575 generic.go:356] "Generic (PLEG): container finished" podID="0d296dcb-2c18-45b4-9ae4-97fc339d9095" containerID="fa2291c7bf028746e715526f472dd9ee1a6dfdabe0829b71a3d08f75b1c61a62" exitCode=1 Oct 30 22:00:57.794359 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794276 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"cada1a26785a64022030d86fb43e9aceff20ca015d9842a5002564d19dfbb10a"} Oct 30 22:00:57.794359 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794351 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"8a7d00a3743b17255be28d270fc0fb67b311a569b15a1034901cf6396f8b4689"} Oct 30 22:00:57.794429 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794404 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"302bfde82564e6f128ac9ac9b648e8633c544dd6cff7dd30322283a8bca9cc49"} Oct 30 22:00:57.794476 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794443 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"041d4dc00968d8d77ea415d9d0d59df0cf6d1046d43410c35acb071a12a5875e"} Oct 30 22:00:57.794519 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794482 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerDied","Data":"fa2291c7bf028746e715526f472dd9ee1a6dfdabe0829b71a3d08f75b1c61a62"} Oct 30 22:00:57.794557 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:57.794528 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"8b80be7b7020bb3bdd8c6eeb6594eef9e6c2af565d6e45cc5eac46518b341ece"} Oct 30 22:00:58.799021 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:58.798990 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="c0a37acde9dd0233fbd744fbf31b269c1f19e3d1579e1a5c96839c68df7d12ed" exitCode=0 Oct 30 22:00:58.799517 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:58.799040 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"c0a37acde9dd0233fbd744fbf31b269c1f19e3d1579e1a5c96839c68df7d12ed"} Oct 30 22:00:59.671443 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.671408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:00:59.671578 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.671504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:00:59.671578 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:59.671521 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:00:59.671578 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.671555 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:00:59.671672 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:59.671630 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:00:59.671720 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:00:59.671689 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:00:59.803975 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.803946 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:00:59.804458 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.804316 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"c064dc085afffa5d220a6b19c93719ebb39d9aef51e256d2db3d8566bc28d133"} Oct 30 22:00:59.807079 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.807052 2575 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="001d42e39c283cb15c5fe1305c7a185d354ec6f5099568b011ac0b4701f611f9" exitCode=0 Oct 30 22:00:59.807179 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:00:59.807098 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerDied","Data":"001d42e39c283cb15c5fe1305c7a185d354ec6f5099568b011ac0b4701f611f9"} Oct 30 22:01:00.814310 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:00.814272 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" event={"ID":"e21847fd-7c43-4822-bc04-a64dba1fc77c","Type":"ContainerStarted","Data":"80920c30eb36a1f00e7d2ba8bb3c693c11bee8f230ea3c73f9c40ef24756e96e"} Oct 30 22:01:00.946457 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:00.946403 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6z8w" podStartSLOduration=7.536292901 podStartE2EDuration="20.94639036s" totalImagesPullingTime="13.410097459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:00.945903626 +0000 UTC m=+21.755995757" watchObservedRunningTime="2026-10-30 22:01:00.94639036 +0000 UTC m=+21.756482521" Oct 30 22:01:01.671049 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.670925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:01.671049 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.670938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:01.671260 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:01.671061 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:01:01.671260 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.671109 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:01.671260 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:01.671186 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:01:01.671260 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:01.671246 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:01:01.819652 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.819625 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:01:01.820146 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.820029 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"32a88f0c30eea843aad70371dc82f3a952ee2fa8458ccd9d966013cca927c3a3"} Oct 30 22:01:01.820498 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:01.820481 2575 scope.go:122] "RemoveContainer" containerID="fa2291c7bf028746e715526f472dd9ee1a6dfdabe0829b71a3d08f75b1c61a62" Oct 30 22:01:02.825487 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.825455 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:01:02.825914 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.825783 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" event={"ID":"0d296dcb-2c18-45b4-9ae4-97fc339d9095","Type":"ContainerStarted","Data":"ce4b2760a272e29086ecdbe2f524db0949630fe93f9b8465357d60c51ecfbb41"} Oct 30 22:01:02.826083 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.826065 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:02.826170 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.826127 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:02.826221 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.826172 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:02.841047 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.841025 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:02.841192 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:02.841148 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:03.118438 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:03.118340 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" podStartSLOduration=8.367267139 podStartE2EDuration="23.11832763s" totalImagesPullingTime="14.751060491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:03.093949035 +0000 UTC m=+23.904041168" watchObservedRunningTime="2026-10-30 22:01:03.11832763 +0000 UTC m=+23.928419760" Oct 30 22:01:03.670591 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:03.670560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:03.670780 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:03.670572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:03.670780 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:03.670686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:03.670780 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:03.670649 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:01:03.670967 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:03.670777 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:01:03.670967 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:03.670884 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:01:04.218277 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.218249 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4j6p"] Oct 30 22:01:04.218736 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.218351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:04.218736 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:04.218498 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:01:04.221205 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.221180 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhvzk"] Oct 30 22:01:04.221329 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.221268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:04.221403 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:04.221343 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:01:04.221814 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.221772 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grfcj"] Oct 30 22:01:04.221905 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:04.221879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:04.221988 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:04.221970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:01:05.671002 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:05.670928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:05.671438 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:05.671026 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:01:05.671438 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:05.670931 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:05.671438 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:05.671071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:05.671438 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:05.671181 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:01:05.671438 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:05.671271 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:01:07.670947 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:07.670918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:07.670947 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:07.670944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:07.671360 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:07.671009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:07.671360 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:07.671103 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4j6p" podUID="91aff955-cd72-4846-b7ec-2e0b63c1056d" Oct 30 22:01:07.671360 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:07.671150 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-grfcj" podUID="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" Oct 30 22:01:07.671360 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:07.671215 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhvzk" podUID="e4971693-7a0b-46a6-80bc-ad5329f213a7" Oct 30 22:01:09.417287 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.417125 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeReady" Oct 30 22:01:09.417617 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.417369 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 22:01:09.671148 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.671076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:09.671309 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.671260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:09.671527 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.671416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:09.728081 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.728056 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 22:01:09.728186 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.728126 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 22:01:09.730621 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.730600 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 22:01:09.780012 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.779988 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\"" Oct 30 22:01:09.780117 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.780016 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 22:01:09.780117 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.780046 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\"" Oct 30 22:01:09.780117 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:09.780074 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\"" Oct 30 22:01:11.499378 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:11.499349 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9hlwl_64ef4db2-e1f8-4784-ad09-f8c612620c64/dns-node-resolver/0.log" Oct 30 22:01:12.721039 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:12.721015 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rnwfr_339dc820-f292-4db9-9ad8-206840a9c6b4/node-ca/0.log" Oct 30 22:01:13.515261 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.515233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:13.515424 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.515276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:13.515424 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:13.515376 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 22:01:13.515494 ip-10-0-134-125 kubenswrapper[2575]: E1030 22:01:13.515437 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs podName:91aff955-cd72-4846-b7ec-2e0b63c1056d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:45.515422193 +0000 UTC m=+66.325514303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs") pod "network-metrics-daemon-m4j6p" (UID: "91aff955-cd72-4846-b7ec-2e0b63c1056d") : secret "metrics-daemon-secret" not found Oct 30 22:01:13.519129 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.519100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:13.605543 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.605506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:13.615615 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.615591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:13.617826 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.617788 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:13.894909 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.894874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:13.983273 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:13.983246 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhvzk"] Oct 30 22:01:13.986338 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:01:13.986304 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4971693_7a0b_46a6_80bc_ad5329f213a7.slice/crio-6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c WatchSource:0}: Error finding container 6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c: Status 404 returned error can't find the container with id 6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c Oct 30 22:01:14.261277 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:14.261254 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-grfcj"] Oct 30 22:01:14.262792 ip-10-0-134-125 kubenswrapper[2575]: W1030 22:01:14.262765 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fd96bb_5a24_4efa_a8f7_e980d8fd5dc8.slice/crio-831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c WatchSource:0}: Error finding container 831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c: Status 404 returned error can't find the container with id 831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c Oct 30 22:01:14.851532 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:14.851495 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grfcj" event={"ID":"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8","Type":"ContainerStarted","Data":"831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c"} Oct 30 22:01:14.852611 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:14.852591 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qhvzk" event={"ID":"e4971693-7a0b-46a6-80bc-ad5329f213a7","Type":"ContainerStarted","Data":"6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c"} Oct 30 22:01:18.864729 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:18.864691 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-grfcj" event={"ID":"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8","Type":"ContainerStarted","Data":"29c3ea09f4acc2633dd29a0a64a8a12629505b0ef4223be19aec6d10cf3246ea"} Oct 30 22:01:18.865194 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:18.864865 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:18.990572 ip-10-0-134-125 kubenswrapper[2575]: I1030 22:01:18.990521 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-grfcj" podStartSLOduration=35.383077683 podStartE2EDuration="38.990508219s" totalImagesPullingTime="3.607430536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:18.990208063 +0000 UTC m=+39.800300185" watchObservedRunningTime="2026-10-30 22:01:18.990508219 +0000 UTC m=+39.800600348" Oct 30 22:01:19.352022 ip-10-0-134-125 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 22:01:19.363186 ip-10-0-134-125 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 22:01:19.363352 ip-10-0-134-125 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 22:01:19.363401 ip-10-0-134-125 systemd[1]: kubelet.service: Consumed 2.123s CPU time. Oct 30 22:01:19.376227 ip-10-0-134-125 systemd[1]: Starting Kubernetes Kubelet... Oct 30 22:01:19.435227 ip-10-0-134-125 kubenswrapper[4305]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:19.435227 ip-10-0-134-125 kubenswrapper[4305]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 22:01:19.435227 ip-10-0-134-125 kubenswrapper[4305]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:19.435227 ip-10-0-134-125 kubenswrapper[4305]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:19.437520 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437446 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.437520 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437515 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.437520 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437520 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.437520 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437525 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437530 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437534 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437538 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437554 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437558 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437563 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437567 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437571 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437574 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437578 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437582 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437586 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437590 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437594 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437598 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437602 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437607 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437611 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.437778 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437615 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437619 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437623 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437627 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437631 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437636 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437640 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437645 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437650 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437656 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437660 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437665 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437669 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437673 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437678 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437682 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437686 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437691 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437695 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437700 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.438681 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437704 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437709 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437713 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437717 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437721 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437729 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437736 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437742 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437746 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437751 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437756 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437760 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437764 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437768 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437773 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437777 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437782 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437786 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437790 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.439584 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437810 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437815 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437819 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437823 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437827 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437831 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437836 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437840 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437844 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437848 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437853 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437873 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437878 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437882 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437886 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437890 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437895 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437899 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437903 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437907 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.440431 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437914 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437919 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437923 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437928 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437933 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437937 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437945 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437952 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437957 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437961 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437967 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437971 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437976 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437980 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437985 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437989 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437993 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.437998 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438002 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.441355 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438006 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438010 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438014 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438019 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438024 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.438036 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439299 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439310 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439314 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439319 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439324 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439329 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439334 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439338 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439342 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439346 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439351 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439355 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439359 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439363 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.442243 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439370 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439374 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439378 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439383 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439387 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439391 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439395 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439399 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439403 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439408 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439413 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439417 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439422 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439426 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439430 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439434 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439438 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439443 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439448 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439452 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.443163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439457 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439461 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439465 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439470 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439474 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439478 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439485 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439489 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439494 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439501 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439508 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439513 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439517 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439521 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439526 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439530 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439534 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439538 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439542 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.443958 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439546 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439550 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439555 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439559 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439564 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439568 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439572 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439577 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439581 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439585 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439589 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439592 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439597 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439602 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439606 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439610 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439615 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439619 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439624 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.444462 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439629 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439633 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439638 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439642 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439646 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439651 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439654 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439661 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439666 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439671 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439676 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439680 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439685 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439689 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439693 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439697 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439701 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439705 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439710 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.444955 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439714 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439719 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439723 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439727 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439731 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439737 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439742 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439747 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439752 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439756 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439761 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439765 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439769 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439773 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.439777 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440558 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440567 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440572 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440576 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440581 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.445491 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440585 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440589 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440593 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440598 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440602 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440606 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440611 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440616 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440620 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440624 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440628 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440633 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440637 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440641 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440645 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440649 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440654 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440658 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440662 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.445997 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440667 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440671 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440675 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440680 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440685 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440689 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440693 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440697 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440701 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440705 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440709 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440714 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440718 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440722 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440727 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440731 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440735 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440739 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440744 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.446498 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440748 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440754 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440760 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440765 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440770 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440774 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440779 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440784 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440788 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440811 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440817 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440822 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440826 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440830 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440836 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440840 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440848 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440854 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440860 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.446970 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440865 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440869 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440874 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440879 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440883 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440888 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440892 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440896 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440901 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440905 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440909 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440913 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440918 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440922 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440926 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440931 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440935 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440939 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440944 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440948 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.447664 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440952 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440956 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440960 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440964 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440969 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440973 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440977 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440982 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440991 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.440995 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441000 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441005 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441009 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441013 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441019 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441024 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441028 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441032 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441037 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441041 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.448333 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441045 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441049 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441053 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.441057 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441201 4305 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441212 4305 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441220 4305 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441228 4305 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441235 4305 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441240 4305 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441246 4305 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441253 4305 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441258 4305 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441262 4305 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441268 4305 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441273 4305 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441278 4305 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441283 4305 flags.go:64] FLAG: --cgroup-root="" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441288 4305 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441293 4305 flags.go:64] FLAG: --client-ca-file="" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441298 4305 flags.go:64] FLAG: --cloud-provider="external" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441303 4305 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441309 4305 flags.go:64] FLAG: --cluster-domain="" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441314 4305 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 22:01:19.448894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441319 4305 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441333 4305 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441339 4305 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441345 4305 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441350 4305 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441356 4305 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441361 4305 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441365 4305 flags.go:64] FLAG: --contention-profiling="false" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441370 4305 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441374 4305 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441379 4305 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441384 4305 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441393 4305 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441399 4305 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441404 4305 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441409 4305 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441414 4305 flags.go:64] FLAG: --enable-server="true" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441418 4305 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441424 4305 flags.go:64] FLAG: --event-burst="100" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441429 4305 flags.go:64] FLAG: --event-qps="50" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441434 4305 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441439 4305 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441444 4305 flags.go:64] FLAG: --eviction-hard="" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441451 4305 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 22:01:19.449496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441455 4305 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441460 4305 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441465 4305 flags.go:64] FLAG: --eviction-soft="" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441470 4305 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441474 4305 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441479 4305 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441485 4305 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441490 4305 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441495 4305 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441504 4305 flags.go:64] FLAG: --feature-gates="" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441511 4305 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441516 4305 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441521 4305 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441526 4305 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441532 4305 flags.go:64] FLAG: --healthz-port="10248" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441537 4305 flags.go:64] FLAG: --help="false" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441542 4305 flags.go:64] FLAG: --hostname-override="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441547 4305 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441552 4305 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441557 4305 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441564 4305 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441569 4305 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441574 4305 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 22:01:19.450102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441579 4305 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441584 4305 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441588 4305 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441593 4305 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441598 4305 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441603 4305 flags.go:64] FLAG: --kube-reserved="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441607 4305 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441612 4305 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441617 4305 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441622 4305 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441627 4305 flags.go:64] FLAG: --lock-file="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441631 4305 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441636 4305 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441641 4305 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441649 4305 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441654 4305 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441660 4305 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441665 4305 flags.go:64] FLAG: --logging-format="text" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441672 4305 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441678 4305 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441682 4305 flags.go:64] FLAG: --manifest-url="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441687 4305 flags.go:64] FLAG: --manifest-url-header="" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441694 4305 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441699 4305 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441705 4305 flags.go:64] FLAG: --max-pods="110" Oct 30 22:01:19.450663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441710 4305 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441715 4305 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441720 4305 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441724 4305 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441731 4305 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441736 4305 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441741 4305 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441751 4305 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441756 4305 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441761 4305 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441766 4305 flags.go:64] FLAG: --pod-cidr="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441771 4305 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441775 4305 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441780 4305 flags.go:64] FLAG: --pods-per-core="0" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441785 4305 flags.go:64] FLAG: --port="10250" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441790 4305 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441812 4305 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0fed0b9b84326d2e7" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441818 4305 flags.go:64] FLAG: --qos-reserved="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441823 4305 flags.go:64] FLAG: --read-only-port="10255" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441828 4305 flags.go:64] FLAG: --register-node="true" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441832 4305 flags.go:64] FLAG: --register-with-taints="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441838 4305 flags.go:64] FLAG: --registry-burst="10" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441842 4305 flags.go:64] FLAG: --registry-qps="5" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441847 4305 flags.go:64] FLAG: --reserved-cpus="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441853 4305 flags.go:64] FLAG: --reserved-memory="" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441861 4305 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 22:01:19.451306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441867 4305 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441872 4305 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441876 4305 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441882 4305 flags.go:64] FLAG: --runonce="false" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441886 4305 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441891 4305 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441896 4305 flags.go:64] FLAG: --seccomp-default="false" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441901 4305 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441905 4305 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441910 4305 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441917 4305 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441922 4305 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441926 4305 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441931 4305 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441935 4305 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441940 4305 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441945 4305 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441951 4305 flags.go:64] FLAG: --system-cgroups="" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441955 4305 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441963 4305 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441968 4305 flags.go:64] FLAG: --tls-cert-file="" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441972 4305 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441979 4305 flags.go:64] FLAG: --tls-min-version="" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441983 4305 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441988 4305 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 22:01:19.452039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441993 4305 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.441998 4305 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442003 4305 flags.go:64] FLAG: --v="2" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442009 4305 flags.go:64] FLAG: --version="false" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442016 4305 flags.go:64] FLAG: --vmodule="" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442022 4305 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442030 4305 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442235 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442242 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442247 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442251 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442255 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442260 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442264 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442269 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442273 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442277 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442284 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442288 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442292 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442297 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442301 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.452661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442306 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442310 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442314 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442318 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442323 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442327 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442332 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442336 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442341 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442346 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442350 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442354 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442358 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442362 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442367 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442371 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442377 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442382 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442387 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442391 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.453254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442395 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442399 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442403 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442408 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442412 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442417 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442421 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442429 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442433 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442438 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442442 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442446 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442450 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442454 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442459 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442463 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442468 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442474 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442481 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.454070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442486 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442490 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442495 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442500 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442504 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442509 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442513 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442518 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442522 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442529 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442533 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442539 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442543 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442548 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442552 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442556 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442561 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442565 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442569 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.454631 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442574 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442579 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442584 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442588 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442593 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442597 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442601 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442606 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442610 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442615 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442619 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442623 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442627 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442632 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442636 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442641 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442645 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442649 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442653 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442657 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.455165 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442662 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442665 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442672 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442677 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442681 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442688 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442694 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442699 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442704 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442708 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442712 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442716 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442720 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.442762 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442834 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442843 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.455861 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442848 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442853 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442857 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442861 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442866 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442870 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442874 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442879 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442883 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442887 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442891 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442896 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442901 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442905 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442909 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442913 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442917 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442922 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442929 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.456326 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442933 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442937 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442942 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442949 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442955 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442960 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442965 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442970 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442974 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442979 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442983 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442990 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.442997 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443001 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443006 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443011 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443015 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443019 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443024 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.457121 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443028 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443032 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443036 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443041 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443045 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443049 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443054 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443058 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443062 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443066 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443070 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443075 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443081 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443085 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443089 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443094 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443125 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443133 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443138 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.458129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443142 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443147 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443152 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443156 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443160 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443166 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443170 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443174 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443178 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443183 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443187 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443193 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443199 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443204 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443208 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443212 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443217 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443221 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443225 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.458927 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443230 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443234 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443238 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443243 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443247 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443252 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443258 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443262 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443267 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443271 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443275 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443279 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443284 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443289 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443293 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443298 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443302 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443306 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443312 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443316 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.459758 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443321 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443325 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443329 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443336 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443340 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443344 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443349 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.443353 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.448345 4305 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.448364 4305 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.448382 4305 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.448387 4305 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448452 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448457 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448461 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448465 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448468 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448471 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448474 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448477 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.460709 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448479 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448482 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448484 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448487 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448490 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448492 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448496 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448499 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448502 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448504 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448507 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448510 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448513 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448516 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448518 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448521 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448524 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448526 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448529 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448531 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.461598 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448534 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448537 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448540 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448543 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448545 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448548 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448551 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448553 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448556 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448559 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448561 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448564 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448566 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448569 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448572 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448574 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448577 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448580 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448583 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.462514 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448587 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448590 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448592 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448595 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448597 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448600 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448603 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448605 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448608 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448611 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448613 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448616 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448618 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448622 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448624 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448627 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448630 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448633 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448635 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448638 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.463419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448641 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448643 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448646 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448648 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448651 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448655 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448659 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448662 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448665 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448668 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448672 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448675 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448679 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448682 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448685 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448688 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448691 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448695 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448698 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.464346 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448701 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448704 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448707 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448710 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448712 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448715 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448718 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448721 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448723 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448726 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448729 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448732 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448735 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448737 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448740 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448743 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448745 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448748 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448750 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.465350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448753 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.448781 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448936 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448942 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448945 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448948 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448951 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448955 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448959 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448963 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448967 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448970 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448973 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448976 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448978 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.466174 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448981 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448983 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448986 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448989 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448992 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448995 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.448998 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449000 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449003 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449006 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449008 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449011 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449013 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449016 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449018 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449021 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449024 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449028 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449032 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.466697 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449035 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449038 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449040 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449044 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449046 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449049 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449052 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449054 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449057 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449060 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449062 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449065 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449067 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449070 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449072 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449075 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449078 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449081 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449084 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449086 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.467296 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449089 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449091 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449094 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449096 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449099 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449101 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449104 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449106 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449109 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449112 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449114 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449117 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449119 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449122 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449124 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449127 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449129 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449132 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449134 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449138 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.468077 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449140 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449143 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449146 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449149 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449151 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449154 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449156 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449159 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449161 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449164 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449167 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449169 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449171 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449174 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449177 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449179 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449182 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449185 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449187 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.468832 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449191 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449193 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449196 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449198 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449201 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449203 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449206 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449208 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449211 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449213 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449216 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449219 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449221 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449224 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449227 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.449250 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:19.469496 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449283 4305 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449288 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449290 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449293 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449296 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449299 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449302 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449304 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449307 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449309 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449312 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449314 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449317 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449320 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449322 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449325 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449335 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449338 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449341 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449344 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:19.470097 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449346 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449349 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449352 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449354 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449357 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449360 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449363 4305 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449366 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449368 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449371 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449374 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449377 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449380 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449383 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449386 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449389 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449391 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449394 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449396 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:19.470585 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449399 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449401 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449404 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449407 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449409 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449412 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449415 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449417 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449420 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449423 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449425 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449428 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449431 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449433 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449436 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449439 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449441 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449444 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449446 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:19.471065 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449449 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449452 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449455 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449458 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449460 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449465 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449468 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449471 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449474 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449477 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449479 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449482 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449484 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449487 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449489 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449495 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449498 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449501 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449504 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:19.471541 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449507 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449510 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449512 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449515 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449518 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449520 4305 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449523 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449526 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449528 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449531 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449535 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449539 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449543 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449546 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449549 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449551 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449554 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449557 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449559 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:19.472047 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449562 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449565 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449567 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449570 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449573 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449575 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449578 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449580 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449584 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:19.449588 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.449664 4305 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.450681 4305 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.450740 4305 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.450941 4305 server.go:1014] "Starting client certificate rotation" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.451079 4305 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.451132 4305 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:40 +0000 UTC" deadline="2028-06-23 04:12:04.979903182 +0000 UTC" Oct 30 22:01:19.472542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.451163 4305 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14430h10m45.528743985s" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.451368 4305 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.452392 4305 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.453854 4305 log.go:25] "Validated CRI v1 runtime API" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.455303 4305 log.go:25] "Validated CRI v1 image API" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.456002 4305 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 22:01:19.472954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.457914 4305 fs.go:135] Filesystem UUIDs: map[677135cd-0124-43ba-b127-2dc3a14f6b76:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 803169c3-f9d2-4c18-aeb3-b0685de0b202:/dev/nvme0n1p3] Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.457934 4305 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~projected/kube-api-access-s6hm6:{mountpoint:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~projected/kube-api-access-s6hm6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/339dc820-f292-4db9-9ad8-206840a9c6b4/volumes/kubernetes.io~projected/kube-api-access-5xqcb:{mountpoint:/var/lib/kubelet/pods/339dc820-f292-4db9-9ad8-206840a9c6b4/volumes/kubernetes.io~projected/kube-api-access-5xqcb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~projected/kube-api-access-m6gcx:{mountpoint:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~projected/kube-api-access-m6gcx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c511576-fd22-4f42-8c20-76c0f1d21464/volumes/kubernetes.io~projected/kube-api-access-x8dpd:{mountpoint:/var/lib/kubelet/pods/4c511576-fd22-4f42-8c20-76c0f1d21464/volumes/kubernetes.io~projected/kube-api-access-x8dpd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b562927-1ada-4f1e-8801-a0ef71f9e654/volumes/kubernetes.io~projected/kube-api-access-m2ht9:{mountpoint:/var/lib/kubelet/pods/5b562927-1ada-4f1e-8801-a0ef71f9e654/volumes/kubernetes.io~projected/kube-api-access-m2ht9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64ef4db2-e1f8-4784-ad09-f8c612620c64/volumes/kubernetes.io~projected/kube-api-access-9clhm:{mountpoint:/var/lib/kubelet/pods/64ef4db2-e1f8-4784-ad09-f8c612620c64/volumes/kubernetes.io~projected/kube-api-access-9clhm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91aff955-cd72-4846-b7ec-2e0b63c1056d/volumes/kubernetes.io~projected/kube-api-access-qc72d:{mountpoint:/var/lib/kubelet/pods/91aff955-cd72-4846-b7ec-2e0b63c1056d/volumes/kubernetes.io~projected/kube-api-access-qc72d major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8/volumes/kubernetes.io~projected/kube-api-access-2zvbx:{mountpoint:/var/lib/kubelet/pods/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8/volumes/kubernetes.io~projected/kube-api-access-2zvbx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2a903ca-0e8b-4fc1-a762-85bf02c32040/volumes/kubernetes.io~projected/kube-api-access-r4hjd:{mountpoint:/var/lib/kubelet/pods/a2a903ca-0e8b-4fc1-a762-85bf02c32040/volumes/kubernetes.io~projected/kube-api-access-r4hjd major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e21847fd-7c43-4822-bc04-a64dba1fc77c/volumes/kubernetes.io~projected/kube-api-access-k295f:{mountpoint:/var/lib/kubelet/pods/e21847fd-7c43-4822-bc04-a64dba1fc77c/volumes/kubernetes.io~projected/kube-api-access-k295f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4971693-7a0b-46a6-80bc-ad5329f213a7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e4971693-7a0b-46a6-80bc-ad5329f213a7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5b7ec02080860e92b0c159502953293728da014ed71553b6c2ca2781d7f3546a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9248dfc4625d68076ea7958565993481f229743bc1f52055eb937eea359589d2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/05187c8779f1b35a2765244a905b6c04ac1c753d518fec2a7dacd0f250755519/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/748a69b29ddcf7282546b5e3831518eaac726b6e03ed7fe08b37edadc73760b6/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/26b7b412611a56fd5f9aaaed3c8f2a58abaacc424e930b0917232fe5fb622b2f/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ddcdafe2c72dd466880da0ed85ffa35c48bb1621b3ae39343769e89270bc9592/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7b61a782fe16f01d9741804ef1a5923721b12928ace20d4378748c5725c12a0a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f7e34e1c7277f3d9ac8123f979388f2d9c678f737f617785271cc73a6d1c197a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/14fd2d0556faf9cc1c2fc8273eaa61d6fae3dab19c0a748bdcb07d0fea6cd0bc/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f39f4547855d8154f2a0adcb86af0604fb3ab8b8a5525183c2bd3be9e0b338ba/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/dac4aefb40d5571b748f371547f8d568cc00d510b877f095b7428d8869e3e887/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6d1e3e59a70331c9a9dcb3dd9f475d0552b49730135a1024890b5a7749407c37/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/890d8329a41acf58663646d6e2920651b54f856fd73380c588ab483cbeeb39a6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4e97271e7eb46bfb0d77fad667964eb8225902441a56e84db3c4d65617a46895/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/148dd29e3d77f2dc514b49b8544005b11cc702dfef23ce502807e4a9f89bc45d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a6cd5b2e5597d2f6bf276ac9da24c698456b247edc9750fdf9c388333ffc6f01/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/28a276d4fd18e289c329a5ba8509242d3c960112db5e51b9d281bfa39d09e89b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3baac79af6a52ee9befb84c93e6484733322bc8d1205cb75fcad03b7dc24907b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/384d964751841979abeae9bbe2ff99e6d30158a980354198f58b319816bbebf4/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/72a6668deadbdca66fe756096d0a2ccfcf169c197f9887314be63e16baa7b765/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0846cef142aee737c3b78cafc36380cb32805a908c89bbd2fe0995d2892b1187/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/55d8f26d44c8ed0dd9a7b82d6485210bc782e348eb3ce7095795eac07aae6c11/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/a9bc23c36c2fa5c821cf56c3fdaed38409ac683043da5389470fc86b25c28411/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/45ea9118f9028ac5614fe727c5711391f998f4409db9c3b91dad47296c858607/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7be0a22fb0cd9546ca9e8b7e77f583438dd27081468456a88d69fadc71d3c071/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f94cd2fa9bf68fa36cd13fb3348e31ecbf82372663efc426080eee62d229f505/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/30a36d0d3e2b82e9c990dd2975a4e4d65239cfdee8f383c503521fb358649919/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7cf559fb9d1ed89ebd7d435b382a9323b60ed8f6119fe3854e8155fcf73a0de7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/add72727f835cb7ef7069fa53e30a6c312b8423b412b979c0f541bcb0c1878b6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/fc85f2abf198b4eec9f7d6e15487e79a7f6f13fd8d687ae75598fc6cb654fda9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f32532630c1dcd9ee60669ecf459a0dbf09e9f3e33d3f624429fe9e6eba8232a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b680411b88d4dae2bbdc73773ca959b12b2edf53dbd6706528fc4d9af135ae04/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e804d0930108d78bcef8fa1a06bd38e9a3e88531fd5176b79d3bc3c8426ed52a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6033438360be38e5835d4bd1f95988317b47c1e8c82b4d375626ed1560cdccc0/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c482872828f73580d17e95eb971b691b779fb65868a3d2fccb3c3a1fc97c845e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.466040 4305 manager.go:217] Machine: {Timestamp:2026-10-30 22:01:19.465238296 +0000 UTC m=+0.060326619 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106501 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e72905c7e9bdf5d2b66e83512936 SystemUUID:ec27e729-05c7-e9bd-f5d2-b66e83512936 BootID:3683f71e-0ee9-4424-8a12-d1eae07ac313 Filesystems:[{Device:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/339dc820-f292-4db9-9ad8-206840a9c6b4/volumes/kubernetes.io~projected/kube-api-access-5xqcb DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2a903ca-0e8b-4fc1-a762-85bf02c32040/volumes/kubernetes.io~projected/kube-api-access-r4hjd DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b562927-1ada-4f1e-8801-a0ef71f9e654/volumes/kubernetes.io~projected/kube-api-access-m2ht9 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb2a7c93ce1e469b255a1f68ee4b36278bb48ab85e568ace4147fe053e8bc783/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62e933e80729a776b4557ef9796516e55967f7bf1496d74f05224b4de974beff/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0428b1654a79bd1f7e5cf46a9849580bac4842bd7f7d231fa17979992d3cb1ec/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5664cd52cc322c5f370b143c4917dcfe8734eb8aee393a8fc4468841305f4b8a/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8/volumes/kubernetes.io~projected/kube-api-access-2zvbx DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/64ef4db2-e1f8-4784-ad09-f8c612620c64/volumes/kubernetes.io~projected/kube-api-access-9clhm DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91aff955-cd72-4846-b7ec-2e0b63c1056d/volumes/kubernetes.io~projected/kube-api-access-qc72d DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5badfc11c64c8da31bd9d025c97bb5db68c2a29fa386d1cf7a34f019a828955f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e21847fd-7c43-4822-bc04-a64dba1fc77c/volumes/kubernetes.io~projected/kube-api-access-k295f DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~projected/kube-api-access-s6hm6 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6907d451cd1477b9a7824bdcb0a321d72a6899a1ca04cdef0ab26f412cf2f94c/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/831a30e8e7e4a5aafc9634dd0c1b5864219c7018f8b1a5b8d574f5f5a3b13d3c/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4780ae8d720da63bcad5d3cfb15cf2dbb6aec3f4f4e457b13ca622a5ac798519/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c59005d20625d22008b389866a0806337496d38ffab488dc37017af150bd6988/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35eeb600ff50257a340e1f331472379e7018b593fb4fbea66d443115282eb5b0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4971693-7a0b-46a6-80bc-ad5329f213a7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0d296dcb-2c18-45b4-9ae4-97fc339d9095/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4c511576-fd22-4f42-8c20-76c0f1d21464/volumes/kubernetes.io~projected/kube-api-access-x8dpd DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f5fbe95-8587-45cc-afab-de9dc77674ba/volumes/kubernetes.io~projected/kube-api-access-m6gcx DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:6907d451cd1477b MacAddress:aa:e4:1d:f4:9c:ee Speed:10000 Mtu:8901} {Name:831a30e8e7e4a5a MacAddress:ba:73:c3:e8:a4:d6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fe:bb:7a:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:4e:05:da:09:72 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:fe:bb:7a:21 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:64:28:0d:ce:79 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:26:d0:63:9c:f5:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.466411 4305 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.466466 4305 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.466761 4305 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.466784 4305 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467072 4305 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467088 4305 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467110 4305 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467131 4305 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467241 4305 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.467770 4305 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468113 4305 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468200 4305 kubelet.go:493] "Attempting to sync node with API server" Oct 30 22:01:19.473126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468228 4305 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468243 4305 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468257 4305 kubelet.go:405] "Adding apiserver pod source" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.468269 4305 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.469405 4305 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.469420 4305 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.469986 4305 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.471725 4305 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.471735 4305 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.471843 4305 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472016 4305 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472042 4305 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472126 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472137 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472144 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472149 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472155 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472161 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472171 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472177 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472184 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472190 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472203 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472213 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472236 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 22:01:19.473714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.472242 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 22:01:19.475443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.475430 4305 server.go:1263] "Started kubelet" Oct 30 22:01:19.475788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.475548 4305 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 22:01:19.475788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.475644 4305 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 22:01:19.475788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.475644 4305 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 22:01:19.476015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.475995 4305 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 22:01:19.476078 ip-10-0-134-125 systemd[1]: Started Kubernetes Kubelet. Oct 30 22:01:19.478242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478225 4305 server.go:317] "Adding debug handlers to kubelet server" Oct 30 22:01:19.478544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478485 4305 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 22:01:19.478544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478498 4305 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 22:01:19.478709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478610 4305 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 22:01:19.478709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478628 4305 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 22:01:19.478883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478739 4305 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 22:01:19.479107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.478978 4305 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:42 +0000 UTC" deadline="2028-06-01 09:15:40.647307677 +0000 UTC" Oct 30 22:01:19.479107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.479046 4305 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13907h14m21.168267723s" Oct 30 22:01:19.479270 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.479237 4305 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:01:19.480186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480153 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b562927-1ada-4f1e-8801-a0ef71f9e654" volumeName="kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480211 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d296dcb-2c18-45b4-9ae4-97fc339d9095" volumeName="kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480229 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d296dcb-2c18-45b4-9ae4-97fc339d9095" volumeName="kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480243 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="339dc820-f292-4db9-9ad8-206840a9c6b4" volumeName="kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480270 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f5fbe95-8587-45cc-afab-de9dc77674ba" volumeName="kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480282 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c511576-fd22-4f42-8c20-76c0f1d21464" volumeName="kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd" seLinuxMountContext="" Oct 30 22:01:19.480307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480295 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b562927-1ada-4f1e-8801-a0ef71f9e654" volumeName="kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480315 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ef4db2-e1f8-4784-ad09-f8c612620c64" volumeName="kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480329 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d296dcb-2c18-45b4-9ae4-97fc339d9095" volumeName="kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480342 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="339dc820-f292-4db9-9ad8-206840a9c6b4" volumeName="kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480377 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c511576-fd22-4f42-8c20-76c0f1d21464" volumeName="kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480391 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b562927-1ada-4f1e-8801-a0ef71f9e654" volumeName="kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480420 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64ef4db2-e1f8-4784-ad09-f8c612620c64" volumeName="kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480442 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91aff955-cd72-4846-b7ec-2e0b63c1056d" volumeName="kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480457 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8" volumeName="kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480469 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2a903ca-0e8b-4fc1-a762-85bf02c32040" volumeName="kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480489 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d296dcb-2c18-45b4-9ae4-97fc339d9095" volumeName="kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480502 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f5fbe95-8587-45cc-afab-de9dc77674ba" volumeName="kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx" seLinuxMountContext="" Oct 30 22:01:19.480559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480516 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3" volumeName="kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.480931 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3" volumeName="kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481016 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21847fd-7c43-4822-bc04-a64dba1fc77c" volumeName="kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481031 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21847fd-7c43-4822-bc04-a64dba1fc77c" volumeName="kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481094 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4971693-7a0b-46a6-80bc-ad5329f213a7" volumeName="kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481108 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d296dcb-2c18-45b4-9ae4-97fc339d9095" volumeName="kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481123 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f5fbe95-8587-45cc-afab-de9dc77674ba" volumeName="kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481170 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21847fd-7c43-4822-bc04-a64dba1fc77c" volumeName="kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481187 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e21847fd-7c43-4822-bc04-a64dba1fc77c" volumeName="kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481205 4305 reconstruct.go:97] "Volume reconstruction finished" Oct 30 22:01:19.481442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.481216 4305 reconciler.go:29] "Reconciler: start to sync state" Oct 30 22:01:19.484441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.484409 4305 factory.go:55] Registering systemd factory Oct 30 22:01:19.484539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.484453 4305 factory.go:223] Registration of the systemd container factory successfully Oct 30 22:01:19.485268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485149 4305 factory.go:153] Registering CRI-O factory Oct 30 22:01:19.485268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485186 4305 factory.go:223] Registration of the crio container factory successfully Oct 30 22:01:19.485268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485262 4305 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 22:01:19.485439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485294 4305 factory.go:103] Registering Raw factory Oct 30 22:01:19.485439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485310 4305 manager.go:1199] Started watching for new ooms in manager Oct 30 22:01:19.485934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.485916 4305 manager.go:319] Starting recovery of all containers Oct 30 22:01:19.494879 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.494852 4305 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 22:01:19.495045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.495026 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:01:19.495601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.495579 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:01:19.510635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.510617 4305 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 22:01:19.510635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.510634 4305 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 22:01:19.510759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.510652 4305 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 22:01:19.510759 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.510684 4305 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 22:01:19.522753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.522733 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:01:19.523120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.523090 4305 generic.go:356] "Generic (PLEG): container finished" podID="0d296dcb-2c18-45b4-9ae4-97fc339d9095" containerID="fa2291c7bf028746e715526f472dd9ee1a6dfdabe0829b71a3d08f75b1c61a62" exitCode=1 Oct 30 22:01:19.528236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528214 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="001d42e39c283cb15c5fe1305c7a185d354ec6f5099568b011ac0b4701f611f9" exitCode=0 Oct 30 22:01:19.528236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528235 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="c0a37acde9dd0233fbd744fbf31b269c1f19e3d1579e1a5c96839c68df7d12ed" exitCode=0 Oct 30 22:01:19.528356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528244 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="f1d3aac38aba77f84168a79058d6c0ec85a796f7cdf0f2cb5460db26763654d8" exitCode=0 Oct 30 22:01:19.528356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528252 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="d9471c9d9858ad40b06fc72d93852c4116529bf0a933b7338085d7f5a9e108b2" exitCode=0 Oct 30 22:01:19.528356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528258 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="a56d4310d9826bb2a1e2d028fc95487e4f866b66320bc08bde846d4ab8c3304b" exitCode=0 Oct 30 22:01:19.528356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.528263 4305 generic.go:356] "Generic (PLEG): container finished" podID="e21847fd-7c43-4822-bc04-a64dba1fc77c" containerID="8b40734068026c85e66c70dc9b9330c890b6412a5b786a249228b3372a668757" exitCode=0 Oct 30 22:01:19.542485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.542456 4305 generic.go:356] "Generic (PLEG): container finished" podID="234561db2c60215956651b28b401629e" containerID="963777689b82e6bfc9ac84800220ad1a55e5ab30342f1be017065199efde3df1" exitCode=0 Oct 30 22:01:19.543811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.543784 4305 manager.go:324] Recovery completed Oct 30 22:01:19.555211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.555196 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.556244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.556228 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.556316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.556256 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.556316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.556268 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.557442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557427 4305 cpu_manager.go:226] "Starting" policy="none" Oct 30 22:01:19.557442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557438 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.557524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557439 4305 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 22:01:19.557524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557506 4305 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 22:01:19.557643 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557627 4305 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 22:01:19.557675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557644 4305 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 22:01:19.557675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557660 4305 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 22:01:19.557675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557666 4305 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 22:01:19.557675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557671 4305 policy_none.go:50] "Start" Oct 30 22:01:19.557778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557681 4305 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 22:01:19.557778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.557691 4305 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 22:01:19.558267 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.558254 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.558318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.558277 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.558318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.558287 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.559472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.559458 4305 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 22:01:19.559505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.559476 4305 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 22:01:19.559505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.559483 4305 policy_none.go:44] "Start" Oct 30 22:01:19.563087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.563062 4305 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 22:01:19.565538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.565523 4305 manager.go:341] "Starting Device Plugin manager" Oct 30 22:01:19.565607 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.565548 4305 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 22:01:19.565607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.565560 4305 server.go:87] "Starting device plugin registration server" Oct 30 22:01:19.565817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.565789 4305 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 22:01:19.565852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.565819 4305 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 22:01:19.565923 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.565907 4305 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 22:01:19.566038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.566005 4305 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 22:01:19.566086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.566047 4305 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 22:01:19.566086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.566054 4305 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 22:01:19.566199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.566125 4305 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:01:19.566037252Z","UUID":"c94a95dc-c2c4-4e86-8a46-47e0d1624b2e","Handler":null,"Name":"","Endpoint":""} Oct 30 22:01:19.567201 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.567181 4305 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 22:01:19.567281 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.567229 4305 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" Oct 30 22:01:19.567526 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.567508 4305 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 22:01:19.567613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.567533 4305 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 22:01:19.611223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.610782 4305 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal"] Oct 30 22:01:19.611311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.611296 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.612089 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612071 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.612141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612102 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.612141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612112 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.612310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612300 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.612441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612426 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.613019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.612998 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.613086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613033 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.613086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613046 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.613086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613046 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.613086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613073 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.613086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613083 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.613228 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.613176 4305 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.613310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613283 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerStarted","Data":"e0f8ebd0c24eab02ff2b40772df168f5db519aacc53466029c47633531b328f2"} Oct 30 22:01:19.613348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613318 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerDied","Data":"963777689b82e6bfc9ac84800220ad1a55e5ab30342f1be017065199efde3df1"} Oct 30 22:01:19.613348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613338 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"234561db2c60215956651b28b401629e","Type":"ContainerStarted","Data":"922ae272c1d0c1dcd26a86afa42ab82043b77d2fc9b691ad8e0290769bb68f82"} Oct 30 22:01:19.613348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613344 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.613438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613351 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"52151281fe9c0d516ce7c27b4aad2f39","Type":"ContainerStarted","Data":"03a494f075f35f2e0994bb9cc02e7d52c68a29bfe0922ebd6d79da256f320468"} Oct 30 22:01:19.613438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.613362 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" event={"ID":"52151281fe9c0d516ce7c27b4aad2f39","Type":"ContainerStarted","Data":"efbb3309439958559b4b5a99274e8d737ccb454eb1f0d75a4310ce1435350be8"} Oct 30 22:01:19.614100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.614087 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.614150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.614110 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.614150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.614120 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.614246 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.614234 4305 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.us-west-2.compute.internal\" not found" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.666056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.666007 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:19.666906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.666890 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:19.666977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.666917 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:19.666977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.666931 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:19.666977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.666952 4305 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.669107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.669089 4305 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 22:01:19.713205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.713180 4305 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:19.713286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.713220 4305 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:19.713346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.713269 4305 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:19.755475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.755456 4305 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:19.769632 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.769610 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.769712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.769655 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.769712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.769680 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.780412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.780393 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.803943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.803926 4305 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.804000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.803982 4305 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.804501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.804488 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeNotReady" Oct 30 22:01:19.804549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.804518 4305 setters.go:546] "Node became not ready" node="ip-10-0-134-125.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T22:01:19Z","lastTransitionTime":"2026-10-30T22:01:19Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 22:01:19.807732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.807714 4305 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-125.us-west-2.compute.internal" is owned by different node (oldNodeID="7f37263f-ea5b-484d-80fd-358380aae86d", newNodeID=""), cleaning up... Oct 30 22:01:19.842597 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.842578 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:19.842658 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.842617 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.842658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.842629 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.869878 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869861 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.869979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869886 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.869979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869905 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.869979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869947 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52151281fe9c0d516ce7c27b4aad2f39-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"52151281fe9c0d516ce7c27b4aad2f39\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.870135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869988 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.870135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.869998 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/234561db2c60215956651b28b401629e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" (UID: \"234561db2c60215956651b28b401629e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:19.895663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:19.895644 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:19.895735 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:19.895679 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:20.069261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.069231 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-125.us-west-2.compute.internal" event="NodeReady" Oct 30 22:01:20.069443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.069312 4305 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 22:01:20.470058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.470024 4305 apiserver.go:52] "Watching apiserver" Oct 30 22:01:20.546101 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.546073 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:20.546257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.546200 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:20.569444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.569419 4305 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 22:01:20.569784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.569766 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-zjfbp","openshift-multus/multus-xwcxh","openshift-multus/network-metrics-daemon-m4j6p","openshift-network-diagnostics/network-check-target-grfcj","openshift-ovn-kubernetes/ovnkube-node-857zd","kube-system/konnectivity-agent-5r47b","kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal","openshift-image-registry/node-ca-rnwfr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-n6z8w","openshift-dns/node-resolver-9hlwl","openshift-network-operator/iptables-alerter-rt97s","kube-system/global-pull-secret-syncer-qhvzk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r"] Oct 30 22:01:20.570663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.570639 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.580463 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.580445 4305 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 22:01:20.604545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.604526 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:20.604638 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:20.604576 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:20.604638 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.604530 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:20.604727 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:20.604645 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.us-west-2.compute.internal" Oct 30 22:01:20.620198 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620162 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\"" Oct 30 22:01:20.620313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620249 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 22:01:20.620313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620249 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\"" Oct 30 22:01:20.620423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620316 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 22:01:20.620423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620332 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\"" Oct 30 22:01:20.620556 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620535 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 22:01:20.620616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620577 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.620616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620577 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.620616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620604 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 22:01:20.620743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620623 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 22:01:20.620743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620626 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.620997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.620985 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 22:01:20.673834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673789 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.673934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673834 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.673934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673861 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.673934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673900 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.673934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673929 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.674147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.673984 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.674147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674058 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.674147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674093 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.674147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674119 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674148 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674171 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674233 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674278 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.674340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674310 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674341 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674366 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-tmp\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674401 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674405 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-cni-binary-copy\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674443 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674485 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674513 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.674598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674575 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674636 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674703 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hjd\" (UniqueName: \"kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674746 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqcb\" (UniqueName: \"kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674775 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674898 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674914 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674927 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674964 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-tuned\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.675023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.674994 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675030 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675063 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675160 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675209 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675249 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675315 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675333 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc72d\" (UniqueName: \"kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675348 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675363 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675425 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675445 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675461 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675480 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675530 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675563 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675669 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.675702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675693 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675726 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.675882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675735 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-config\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.675882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675750 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:01:20.675882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.675787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.676156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676132 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.676211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676177 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676204 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676139 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64ef4db2-e1f8-4784-ad09-f8c612620c64-tmp-dir\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.676307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676234 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676328 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.676391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676377 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676403 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676438 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gcx\" (UniqueName: \"kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676499 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k295f\" (UniqueName: \"kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.676562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676536 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.676647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676561 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.676647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676592 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.676647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676622 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676653 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676687 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676716 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676740 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676772 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676815 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676848 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dpd\" (UniqueName: \"kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676877 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676902 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676931 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.676998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.676964 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677000 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677029 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677057 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677090 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clhm\" (UniqueName: \"kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677129 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hm6\" (UniqueName: \"kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677157 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ht9\" (UniqueName: \"kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.677225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.677181 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.678356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.678325 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovn-node-metrics-cert\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.679221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679191 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e21847fd-7c43-4822-bc04-a64dba1fc77c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.679899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679872 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679905 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679971 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\"" Oct 30 22:01:20.680002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679981 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.679979 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680032 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680037 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680077 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680095 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680105 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680139 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680153 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680166 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680190 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680191 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680206 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680219 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680227 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680253 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 22:01:20.680282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680259 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680305 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680300 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680394 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680448 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680508 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680537 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680572 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680587 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680641 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680684 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\"" Oct 30 22:01:20.680925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.680880 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 22:01:20.684672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.684656 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-env-overrides\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.684780 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.684766 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4c511576-fd22-4f42-8c20-76c0f1d21464-iptables-alerter-script\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.685892 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.685873 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/339dc820-f292-4db9-9ad8-206840a9c6b4-serviceca\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.685949 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.685938 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-konnectivity-ca\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:01:20.689102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.689083 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e4971693-7a0b-46a6-80bc-ad5329f213a7-original-pull-secret\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.689175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.689083 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3-agent-certs\") pod \"konnectivity-agent-5r47b\" (UID: \"b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3\") " pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:01:20.689175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.689115 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-daemon-config\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.689175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.689142 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d296dcb-2c18-45b4-9ae4-97fc339d9095-ovnkube-script-lib\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.734186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.734109 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hjd\" (UniqueName: \"kubernetes.io/projected/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kube-api-access-r4hjd\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.734186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.734151 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc72d\" (UniqueName: \"kubernetes.io/projected/91aff955-cd72-4846-b7ec-2e0b63c1056d-kube-api-access-qc72d\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.734186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.734169 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqcb\" (UniqueName: \"kubernetes.io/projected/339dc820-f292-4db9-9ad8-206840a9c6b4-kube-api-access-5xqcb\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.734345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.734235 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvbx\" (UniqueName: \"kubernetes.io/projected/91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8-kube-api-access-2zvbx\") pod \"network-check-target-grfcj\" (UID: \"91fd96bb-5a24-4efa-a8f7-e980d8fd5dc8\") " pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:20.735126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.735107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dpd\" (UniqueName: \"kubernetes.io/projected/4c511576-fd22-4f42-8c20-76c0f1d21464-kube-api-access-x8dpd\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.735465 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.735444 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k295f\" (UniqueName: \"kubernetes.io/projected/e21847fd-7c43-4822-bc04-a64dba1fc77c-kube-api-access-k295f\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.735660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.735638 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clhm\" (UniqueName: \"kubernetes.io/projected/64ef4db2-e1f8-4784-ad09-f8c612620c64-kube-api-access-9clhm\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.735850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.735660 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hm6\" (UniqueName: \"kubernetes.io/projected/0d296dcb-2c18-45b4-9ae4-97fc339d9095-kube-api-access-s6hm6\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.736105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.736086 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gcx\" (UniqueName: \"kubernetes.io/projected/3f5fbe95-8587-45cc-afab-de9dc77674ba-kube-api-access-m6gcx\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.740203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.740183 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ht9\" (UniqueName: \"kubernetes.io/projected/5b562927-1ada-4f1e-8801-a0ef71f9e654-kube-api-access-m2ht9\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.777736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777712 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.777847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777740 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.777847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777755 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.777847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777775 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.777847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777809 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.777847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777849 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777858 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-ovn\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777869 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-node-log\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777862 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-sys-fs\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777875 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777887 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-netd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777912 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-registration-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-systemd-units\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777926 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-systemd\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777951 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777955 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-conf\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777978 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-systemd\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.777982 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778012 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778039 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778063 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778067 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778102 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-dbus\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.778098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778111 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778136 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778149 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e4971693-7a0b-46a6-80bc-ad5329f213a7-kubelet-config\") pod \"global-pull-secret-syncer-qhvzk\" (UID: \"e4971693-7a0b-46a6-80bc-ad5329f213a7\") " pod="kube-system/global-pull-secret-syncer-qhvzk" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-multus-certs\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778138 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-kubelet\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778160 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778191 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778205 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-sys\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778212 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778220 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysctl-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778228 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778246 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-k8s-cni-cncf-io\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778247 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c511576-fd22-4f42-8c20-76c0f1d21464-host-slash\") pod \"iptables-alerter-rt97s\" (UID: \"4c511576-fd22-4f42-8c20-76c0f1d21464\") " pod="openshift-network-operator/iptables-alerter-rt97s" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778257 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778281 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778292 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-cni-bin\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778297 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-system-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778301 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778332 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778349 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778351 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-socket-dir-parent\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778364 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778386 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-run\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778390 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778402 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-system-cni-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778413 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-var-lib-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778407 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/339dc820-f292-4db9-9ad8-206840a9c6b4-host\") pod \"node-ca-rnwfr\" (UID: \"339dc820-f292-4db9-9ad8-206840a9c6b4\") " pod="openshift-image-registry/node-ca-rnwfr" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778424 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778446 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-run-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778451 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778454 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64ef4db2-e1f8-4784-ad09-f8c612620c64-hosts-file\") pod \"node-resolver-9hlwl\" (UID: \"64ef4db2-e1f8-4784-ad09-f8c612620c64\") " pod="openshift-dns/node-resolver-9hlwl" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778490 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778518 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-modprobe-d\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778550 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778563 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-var-lib-kubelet\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778579 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778580 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-device-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778607 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-slash\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778660 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778685 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-hostroot\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778687 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-socket-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778692 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-run-netns\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778708 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778751 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778773 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778811 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-conf-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778822 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-multus\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778829 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-multus-cni-dir\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778830 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778857 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-etc-kubernetes\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778862 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-log-socket\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778869 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778927 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-cnibin\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778929 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-ovn-kubernetes\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778951 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778966 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778986 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e21847fd-7c43-4822-bc04-a64dba1fc77c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6z8w\" (UID: \"e21847fd-7c43-4822-bc04-a64dba1fc77c\") " pod="openshift-multus/multus-additional-cni-plugins-n6z8w" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778964 4305 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.778994 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779035 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-cnibin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779013 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-kubelet\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779043 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779071 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-kubernetes\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779069 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779096 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.779047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779103 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b562927-1ada-4f1e-8801-a0ef71f9e654-host-var-lib-cni-bin\") pod \"multus-xwcxh\" (UID: \"5b562927-1ada-4f1e-8801-a0ef71f9e654\") " pod="openshift-multus/multus-xwcxh" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779113 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779127 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779145 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-host-run-netns\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779162 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-etc-sysconfig\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779195 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-lib-modules\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779226 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779232 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d296dcb-2c18-45b4-9ae4-97fc339d9095-etc-openvswitch\") pod \"ovnkube-node-857zd\" (UID: \"0d296dcb-2c18-45b4-9ae4-97fc339d9095\") " pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779269 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f5fbe95-8587-45cc-afab-de9dc77674ba-host\") pod \"tuned-zjfbp\" (UID: \"3f5fbe95-8587-45cc-afab-de9dc77674ba\") " pod="openshift-cluster-node-tuning-operator/tuned-zjfbp" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.779269 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a2a903ca-0e8b-4fc1-a762-85bf02c32040-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6b8r\" (UID: \"a2a903ca-0e8b-4fc1-a762-85bf02c32040\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6b8r" Oct 30 22:01:20.781187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.780964 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aff955-cd72-4846-b7ec-2e0b63c1056d-metrics-certs\") pod \"network-metrics-daemon-m4j6p\" (UID: \"91aff955-cd72-4846-b7ec-2e0b63c1056d\") " pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:20.871120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:20.871084 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4j6p" Oct 30 22:01:21.235171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:21.235143 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4j6p"] Oct 30 22:01:21.242901 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:21.242875 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aff955_cd72_4846_b7ec_2e0b63c1056d.slice/crio-426de14a57699aed4d7e2194e26e3c60eba8577dfe4b7555cf220bd25182c1c8 WatchSource:0}: Error finding container 426de14a57699aed4d7e2194e26e3c60eba8577dfe4b7555cf220bd25182c1c8: Status 404 returned error can't find the container with id 426de14a57699aed4d7e2194e26e3c60eba8577dfe4b7555cf220bd25182c1c8 Oct 30 22:01:21.245553 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:21.245533 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:01:21.549304 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:21.549222 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4j6p" event={"ID":"91aff955-cd72-4846-b7ec-2e0b63c1056d","Type":"ContainerStarted","Data":"426de14a57699aed4d7e2194e26e3c60eba8577dfe4b7555cf220bd25182c1c8"} Oct 30 22:01:22.876607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:22.876580 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:23.556316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:23.556285 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4j6p" event={"ID":"91aff955-cd72-4846-b7ec-2e0b63c1056d","Type":"ContainerStarted","Data":"18b29f242bd77622054b6489b884517ee5ae30892d6f85ed8193aa77a144cc0d"} Oct 30 22:01:23.556477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:23.556344 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4j6p" event={"ID":"91aff955-cd72-4846-b7ec-2e0b63c1056d","Type":"ContainerStarted","Data":"1fdce06e6f6d21237b4e7f1536998b29ae771083b9b78190a7b7c33a43b8542f"} Oct 30 22:01:25.120874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:25.120839 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5r47b" Oct 30 22:01:26.685318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.685290 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j"] Oct 30 22:01:26.772141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.772111 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7d49"] Oct 30 22:01:26.772290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.772251 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:26.784657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.784632 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mjq4c"] Oct 30 22:01:26.784773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.784743 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.798806 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.798763 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hsrj5"] Oct 30 22:01:26.798917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.798833 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.810030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810002 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b894-7e30-4650-9b98-66d92ba11809-tmp-dir\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.810170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810047 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b488d9e4-12ad-4df4-9cda-25839d4620f4-tmp\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.810170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810132 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f55dcc39-82dd-4403-bee1-38b004c540d7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8cz8j\" (UID: \"f55dcc39-82dd-4403-bee1-38b004c540d7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:26.810279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810256 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lp8\" (UniqueName: \"kubernetes.io/projected/4e68b894-7e30-4650-9b98-66d92ba11809-kube-api-access-n2lp8\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.810320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810282 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wzm\" (UniqueName: \"kubernetes.io/projected/b488d9e4-12ad-4df4-9cda-25839d4620f4-kube-api-access-s8wzm\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.810320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810312 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e68b894-7e30-4650-9b98-66d92ba11809-metrics-tls\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.810411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.810340 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e68b894-7e30-4650-9b98-66d92ba11809-config-volume\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.820738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.820713 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-26pqc"] Oct 30 22:01:26.820861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.820846 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:26.824645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.824627 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sh8j4\"" Oct 30 22:01:26.824732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.824721 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 22:01:26.833613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.833592 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j"] Oct 30 22:01:26.833714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.833628 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7d49"] Oct 30 22:01:26.833714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.833656 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mjq4c"] Oct 30 22:01:26.833714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.833678 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsrj5"] Oct 30 22:01:26.833714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.833707 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26pqc"] Oct 30 22:01:26.834113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.834097 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:26.835846 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.835821 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 22:01:26.849444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.849423 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 22:01:26.849536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.849501 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 22:01:26.869913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.869891 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 22:01:26.870025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.869931 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 22:01:26.879744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.879726 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 22:01:26.884510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.884496 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 22:01:26.884590 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.884577 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 22:01:26.910965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.910934 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lp8\" (UniqueName: \"kubernetes.io/projected/4e68b894-7e30-4650-9b98-66d92ba11809-kube-api-access-n2lp8\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.910965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.910970 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wzm\" (UniqueName: \"kubernetes.io/projected/b488d9e4-12ad-4df4-9cda-25839d4620f4-kube-api-access-s8wzm\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.911174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.910993 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e68b894-7e30-4650-9b98-66d92ba11809-metrics-tls\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.911174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911015 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e68b894-7e30-4650-9b98-66d92ba11809-config-volume\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.911174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911030 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b894-7e30-4650-9b98-66d92ba11809-tmp-dir\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.911174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911052 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b488d9e4-12ad-4df4-9cda-25839d4620f4-tmp\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.911174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911144 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f55dcc39-82dd-4403-bee1-38b004c540d7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8cz8j\" (UID: \"f55dcc39-82dd-4403-bee1-38b004c540d7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:26.911434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911414 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e68b894-7e30-4650-9b98-66d92ba11809-tmp-dir\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.911473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911443 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b488d9e4-12ad-4df4-9cda-25839d4620f4-tmp\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:26.911640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.911614 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e68b894-7e30-4650-9b98-66d92ba11809-config-volume\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.913443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.913425 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f55dcc39-82dd-4403-bee1-38b004c540d7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8cz8j\" (UID: \"f55dcc39-82dd-4403-bee1-38b004c540d7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:26.924775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924750 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 22:01:26.924888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924837 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 22:01:26.924888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924844 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zzmb5\"" Oct 30 22:01:26.924979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924967 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd25c\"" Oct 30 22:01:26.925031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924983 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqg6l\"" Oct 30 22:01:26.925031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.924993 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 22:01:26.925031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.925013 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g5t42\"" Oct 30 22:01:26.933404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.933383 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e68b894-7e30-4650-9b98-66d92ba11809-metrics-tls\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.971823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.971785 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lp8\" (UniqueName: \"kubernetes.io/projected/4e68b894-7e30-4650-9b98-66d92ba11809-kube-api-access-n2lp8\") pod \"dns-default-d7d49\" (UID: \"4e68b894-7e30-4650-9b98-66d92ba11809\") " pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:26.971952 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:26.971857 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wzm\" (UniqueName: \"kubernetes.io/projected/b488d9e4-12ad-4df4-9cda-25839d4620f4-kube-api-access-s8wzm\") pod \"downloads-6dd789c8b7-mjq4c\" (UID: \"b488d9e4-12ad-4df4-9cda-25839d4620f4\") " pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:27.012292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012262 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a329bcb5-72e6-4a32-8162-b3239348740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.012464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012301 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-cert\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.012464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012325 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmcr\" (UniqueName: \"kubernetes.io/projected/a329bcb5-72e6-4a32-8162-b3239348740d-kube-api-access-xnmcr\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.012464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012387 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7k2\" (UniqueName: \"kubernetes.io/projected/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-kube-api-access-tl7k2\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.012568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012502 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a329bcb5-72e6-4a32-8162-b3239348740d-crio-socket\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.012568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012533 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a329bcb5-72e6-4a32-8162-b3239348740d-data-volume\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.012568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.012550 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a329bcb5-72e6-4a32-8162-b3239348740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.093786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.093752 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:27.113321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113291 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a329bcb5-72e6-4a32-8162-b3239348740d-crio-socket\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113323 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a329bcb5-72e6-4a32-8162-b3239348740d-data-volume\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113347 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a329bcb5-72e6-4a32-8162-b3239348740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113376 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a329bcb5-72e6-4a32-8162-b3239348740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113420 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a329bcb5-72e6-4a32-8162-b3239348740d-crio-socket\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113481 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-cert\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.113533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113522 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmcr\" (UniqueName: \"kubernetes.io/projected/a329bcb5-72e6-4a32-8162-b3239348740d-kube-api-access-xnmcr\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113545 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7k2\" (UniqueName: \"kubernetes.io/projected/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-kube-api-access-tl7k2\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.113757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113744 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a329bcb5-72e6-4a32-8162-b3239348740d-data-volume\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.113958 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.113939 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a329bcb5-72e6-4a32-8162-b3239348740d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.115823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.115791 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a329bcb5-72e6-4a32-8162-b3239348740d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.115931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.115890 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-cert\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.126673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.126658 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:27.147827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.147785 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:27.172377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.172329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmcr\" (UniqueName: \"kubernetes.io/projected/a329bcb5-72e6-4a32-8162-b3239348740d-kube-api-access-xnmcr\") pod \"insights-runtime-extractor-hsrj5\" (UID: \"a329bcb5-72e6-4a32-8162-b3239348740d\") " pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.172554 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.172424 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7k2\" (UniqueName: \"kubernetes.io/projected/6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4-kube-api-access-tl7k2\") pod \"ingress-canary-26pqc\" (UID: \"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4\") " pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.202170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.202142 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26pqc" Oct 30 22:01:27.223258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.223191 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hsrj5" Oct 30 22:01:27.484023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.483956 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j"] Oct 30 22:01:27.491034 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:27.491008 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55dcc39_82dd_4403_bee1_38b004c540d7.slice/crio-7974cff78ffda23f7a750c1006ee8436b97fea85bb197abe4c8f23644559057b WatchSource:0}: Error finding container 7974cff78ffda23f7a750c1006ee8436b97fea85bb197abe4c8f23644559057b: Status 404 returned error can't find the container with id 7974cff78ffda23f7a750c1006ee8436b97fea85bb197abe4c8f23644559057b Oct 30 22:01:27.535479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.535455 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7d49"] Oct 30 22:01:27.543213 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:27.543190 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e68b894_7e30_4650_9b98_66d92ba11809.slice/crio-b372b57ecb6e3668ee591f04ab54f4928b0caf4aa00e38a1e104c64c91e511a7 WatchSource:0}: Error finding container b372b57ecb6e3668ee591f04ab54f4928b0caf4aa00e38a1e104c64c91e511a7: Status 404 returned error can't find the container with id b372b57ecb6e3668ee591f04ab54f4928b0caf4aa00e38a1e104c64c91e511a7 Oct 30 22:01:27.566411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.566383 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7d49" event={"ID":"4e68b894-7e30-4650-9b98-66d92ba11809","Type":"ContainerStarted","Data":"b372b57ecb6e3668ee591f04ab54f4928b0caf4aa00e38a1e104c64c91e511a7"} Oct 30 22:01:27.567416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.567395 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" event={"ID":"f55dcc39-82dd-4403-bee1-38b004c540d7","Type":"ContainerStarted","Data":"7974cff78ffda23f7a750c1006ee8436b97fea85bb197abe4c8f23644559057b"} Oct 30 22:01:27.576792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.576775 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mjq4c"] Oct 30 22:01:27.583710 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:27.583689 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb488d9e4_12ad_4df4_9cda_25839d4620f4.slice/crio-3a4beff65a6a10b0997c906c61c689c4bf7d7c6c0b57be0690afd246f7628973 WatchSource:0}: Error finding container 3a4beff65a6a10b0997c906c61c689c4bf7d7c6c0b57be0690afd246f7628973: Status 404 returned error can't find the container with id 3a4beff65a6a10b0997c906c61c689c4bf7d7c6c0b57be0690afd246f7628973 Oct 30 22:01:27.618622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.618602 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26pqc"] Oct 30 22:01:27.626240 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:27.626217 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a96a37a_5e64_4ac8_bb0f_c6bc27aaffc4.slice/crio-69a0072c7d1862adace8281d3e61d3b84f43c2f009d7a5b31d9110d8865f3ef4 WatchSource:0}: Error finding container 69a0072c7d1862adace8281d3e61d3b84f43c2f009d7a5b31d9110d8865f3ef4: Status 404 returned error can't find the container with id 69a0072c7d1862adace8281d3e61d3b84f43c2f009d7a5b31d9110d8865f3ef4 Oct 30 22:01:27.644843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:27.644816 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hsrj5"] Oct 30 22:01:27.651661 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:27.651640 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda329bcb5_72e6_4a32_8162_b3239348740d.slice/crio-302e08aa0c6976bf17d7b1704a70644b28d0f1606a9690bc463df91da51c8112 WatchSource:0}: Error finding container 302e08aa0c6976bf17d7b1704a70644b28d0f1606a9690bc463df91da51c8112: Status 404 returned error can't find the container with id 302e08aa0c6976bf17d7b1704a70644b28d0f1606a9690bc463df91da51c8112 Oct 30 22:01:28.507136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:28.507083 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:28.571214 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:28.571179 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mjq4c" event={"ID":"b488d9e4-12ad-4df4-9cda-25839d4620f4","Type":"ContainerStarted","Data":"3a4beff65a6a10b0997c906c61c689c4bf7d7c6c0b57be0690afd246f7628973"} Oct 30 22:01:28.572709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:28.572681 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsrj5" event={"ID":"a329bcb5-72e6-4a32-8162-b3239348740d","Type":"ContainerStarted","Data":"2fa3680ccffdfca3313ec87cce6f9047011fd3b482412b5869fcc37314ffc6d2"} Oct 30 22:01:28.572843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:28.572736 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsrj5" event={"ID":"a329bcb5-72e6-4a32-8162-b3239348740d","Type":"ContainerStarted","Data":"302e08aa0c6976bf17d7b1704a70644b28d0f1606a9690bc463df91da51c8112"} Oct 30 22:01:28.573781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:28.573758 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26pqc" event={"ID":"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4","Type":"ContainerStarted","Data":"69a0072c7d1862adace8281d3e61d3b84f43c2f009d7a5b31d9110d8865f3ef4"} Oct 30 22:01:30.592396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:30.592311 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsrj5" event={"ID":"a329bcb5-72e6-4a32-8162-b3239348740d","Type":"ContainerStarted","Data":"377390c54cb89b9eb6f9a19dab2b602424791d73e081a794c48da6b28491e2f5"} Oct 30 22:01:30.594843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:30.594783 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" event={"ID":"f55dcc39-82dd-4403-bee1-38b004c540d7","Type":"ContainerStarted","Data":"c6a7af8e1d9d34324e28e785117cf42506529f891cbd1b028383f2aed530b216"} Oct 30 22:01:30.704854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:30.704739 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" podStartSLOduration=2.368568409 podStartE2EDuration="4.704721096s" totalImagesPullingTime="2.336152687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:30.704372203 +0000 UTC m=+11.299460509" watchObservedRunningTime="2026-10-30 22:01:30.704721096 +0000 UTC m=+11.299809401" Oct 30 22:01:30.721874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:30.721841 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:30.722108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:30.722092 4305 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:01:31.600871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:31.600832 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7d49" event={"ID":"4e68b894-7e30-4650-9b98-66d92ba11809","Type":"ContainerStarted","Data":"0e8c402fb8888722cbc076a5e2c403df443d2c85be1131cdedff750503e6d449"} Oct 30 22:01:31.601351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:31.600892 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7d49" event={"ID":"4e68b894-7e30-4650-9b98-66d92ba11809","Type":"ContainerStarted","Data":"69fb6f8a4307c29f8f70093d76718242866dec5dbb6c32f4817ed68186dca12b"} Oct 30 22:01:31.601351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:31.600854 4305 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:01:31.601351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:31.601082 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:31.715238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:31.715185 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-d7d49" podStartSLOduration=2.437194872 podStartE2EDuration="5.715168646s" totalImagesPullingTime="3.277973774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:31.711629306 +0000 UTC m=+12.306717616" watchObservedRunningTime="2026-10-30 22:01:31.715168646 +0000 UTC m=+12.310256953" Oct 30 22:01:32.606868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:32.606831 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26pqc" event={"ID":"6a96a37a-5e64-4ac8-bb0f-c6bc27aaffc4","Type":"ContainerStarted","Data":"5a820890b702caa195bd19414a7754f1489f5717f732f9219a32a5c4e5982ffc"} Oct 30 22:01:33.613579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:33.613536 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hsrj5" event={"ID":"a329bcb5-72e6-4a32-8162-b3239348740d","Type":"ContainerStarted","Data":"1496993d9ab5a481e40f8feec670cd004f70a9227faaa3bb8c109a7e82a5cc5b"} Oct 30 22:01:33.725861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:33.725813 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hsrj5" podStartSLOduration=2.685219687 podStartE2EDuration="7.725785881s" totalImagesPullingTime="5.040566194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:33.725073879 +0000 UTC m=+14.320162187" watchObservedRunningTime="2026-10-30 22:01:33.725785881 +0000 UTC m=+14.320874185" Oct 30 22:01:33.726023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:33.725972 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-26pqc" podStartSLOduration=3.268498182 podStartE2EDuration="7.725966672s" totalImagesPullingTime="4.45746849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:32.715931703 +0000 UTC m=+13.311020009" watchObservedRunningTime="2026-10-30 22:01:33.725966672 +0000 UTC m=+14.321054972" Oct 30 22:01:34.693550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:34.693513 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:34.693980 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:34.693710 4305 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:01:34.699276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:34.699253 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8cz8j" Oct 30 22:01:36.166738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.166703 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg"] Oct 30 22:01:36.169223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.169200 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.172410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.172383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.172521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.172415 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54829c0-8734-4647-9589-b9046fbe56a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.172521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.172431 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb682\" (UniqueName: \"kubernetes.io/projected/a54829c0-8734-4647-9589-b9046fbe56a5-kube-api-access-rb682\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.172521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.172494 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.264376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264349 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-klql5\"" Oct 30 22:01:36.264376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264346 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 22:01:36.264596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264488 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 22:01:36.264596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264518 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 22:01:36.264596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264570 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 22:01:36.264686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.264650 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 22:01:36.273004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.272984 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.273133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.273017 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.273133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.273047 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54829c0-8734-4647-9589-b9046fbe56a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.273133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.273066 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb682\" (UniqueName: \"kubernetes.io/projected/a54829c0-8734-4647-9589-b9046fbe56a5-kube-api-access-rb682\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.273695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.273674 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54829c0-8734-4647-9589-b9046fbe56a5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.275682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.275656 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.275904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.275887 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a54829c0-8734-4647-9589-b9046fbe56a5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.311413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.311351 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg"] Oct 30 22:01:36.330893 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.330869 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb682\" (UniqueName: \"kubernetes.io/projected/a54829c0-8734-4647-9589-b9046fbe56a5-kube-api-access-rb682\") pod \"prometheus-operator-9cd89b74c-r2jvg\" (UID: \"a54829c0-8734-4647-9589-b9046fbe56a5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.482714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.482683 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" Oct 30 22:01:36.493248 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.493222 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:01:36.495527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.495508 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547110 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547254 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547318 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547400 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547448 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 22:01:36.547823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.547513 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pw72t\"" Oct 30 22:01:36.556370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.556294 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:01:36.676294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676264 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676300 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676339 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676410 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676427 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.676637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.676446 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2t9\" (UniqueName: \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777600 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777637 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777653 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777669 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2t9\" (UniqueName: \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777706 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.777983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.777728 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.778495 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.778460 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.778619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.778476 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.778619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.778571 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.779096 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.779075 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.780266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.780233 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.780993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.780965 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.836300 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.836277 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2t9\" (UniqueName: \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\") pod \"console-66bd6b67db-9bzrk\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:36.854817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:36.854779 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg"] Oct 30 22:01:36.864846 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:36.864818 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54829c0_8734_4647_9589_b9046fbe56a5.slice/crio-56619125ffe572b1538d6c650922fa52855f6b06ec9058b77753af6033d6885a WatchSource:0}: Error finding container 56619125ffe572b1538d6c650922fa52855f6b06ec9058b77753af6033d6885a: Status 404 returned error can't find the container with id 56619125ffe572b1538d6c650922fa52855f6b06ec9058b77753af6033d6885a Oct 30 22:01:37.113575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:37.113500 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:37.491276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:37.491244 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:01:37.526653 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:37.526624 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f591f45_fb13_4a4e_b5dd_3e7eb63e7a3e.slice/crio-16ce51e222e6b1f46a16020e1717550a62ed8ecd61af90a5961c84d14c20b94e WatchSource:0}: Error finding container 16ce51e222e6b1f46a16020e1717550a62ed8ecd61af90a5961c84d14c20b94e: Status 404 returned error can't find the container with id 16ce51e222e6b1f46a16020e1717550a62ed8ecd61af90a5961c84d14c20b94e Oct 30 22:01:37.628703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:37.628668 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" event={"ID":"a54829c0-8734-4647-9589-b9046fbe56a5","Type":"ContainerStarted","Data":"56619125ffe572b1538d6c650922fa52855f6b06ec9058b77753af6033d6885a"} Oct 30 22:01:37.630108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:37.630079 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bd6b67db-9bzrk" event={"ID":"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e","Type":"ContainerStarted","Data":"16ce51e222e6b1f46a16020e1717550a62ed8ecd61af90a5961c84d14c20b94e"} Oct 30 22:01:39.639516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:39.639462 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" event={"ID":"a54829c0-8734-4647-9589-b9046fbe56a5","Type":"ContainerStarted","Data":"4ee0358479699d68cf46c683cb38554952b909c672fcf8a0006c032d22660689"} Oct 30 22:01:39.640169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:39.639544 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" event={"ID":"a54829c0-8734-4647-9589-b9046fbe56a5","Type":"ContainerStarted","Data":"cc57c78a34f916879de63f8c60a895a6f4e5b5e978dd034fdcfe3bc3e2eb5192"} Oct 30 22:01:39.754362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:39.754305 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r2jvg" podStartSLOduration=1.7626353529999998 podStartE2EDuration="3.754292105s" totalImagesPullingTime="1.991656752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:39.753241671 +0000 UTC m=+20.348329977" watchObservedRunningTime="2026-10-30 22:01:39.754292105 +0000 UTC m=+20.349380411" Oct 30 22:01:41.609571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:41.609537 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7d49" Oct 30 22:01:41.650571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:41.650526 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bd6b67db-9bzrk" event={"ID":"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e","Type":"ContainerStarted","Data":"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479"} Oct 30 22:01:41.836519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:41.836457 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66bd6b67db-9bzrk" podStartSLOduration=2.383136188 podStartE2EDuration="5.836438273s" totalImagesPullingTime="3.453302085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:41.835912026 +0000 UTC m=+22.431000331" watchObservedRunningTime="2026-10-30 22:01:41.836438273 +0000 UTC m=+22.431526583" Oct 30 22:01:42.529150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.529114 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn"] Oct 30 22:01:42.531473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.531452 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.580974 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.580946 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 22:01:42.620365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.620331 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebf74d8f-5bae-4c22-851c-564d30ef8088-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.620788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.620392 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.620788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.620528 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.620788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.620561 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2r7\" (UniqueName: \"kubernetes.io/projected/ebf74d8f-5bae-4c22-851c-564d30ef8088-kube-api-access-6q2r7\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.626466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.626441 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rnpmb\"" Oct 30 22:01:42.626591 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.626503 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 22:01:42.627471 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.627450 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn"] Oct 30 22:01:42.674435 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.674407 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86"] Oct 30 22:01:42.676728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.676710 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cg66d"] Oct 30 22:01:42.676847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.676834 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.679689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.679671 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.721941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.721912 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.721956 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.721975 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hbl\" (UniqueName: \"kubernetes.io/projected/6d2c6300-9238-44b0-910b-ce103e181d4a-kube-api-access-f4hbl\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722045 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2r7\" (UniqueName: \"kubernetes.io/projected/ebf74d8f-5bae-4c22-851c-564d30ef8088-kube-api-access-6q2r7\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.722249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722230 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-sys\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722269 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-textfile\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722302 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.722407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722332 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-root\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722363 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebf74d8f-5bae-4c22-851c-564d30ef8088-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.722502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722419 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.722550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722501 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.722588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722547 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-wtmp\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722593 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lr2m\" (UniqueName: \"kubernetes.io/projected/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-api-access-8lr2m\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.722689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722651 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.722744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.722814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722759 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-metrics-client-ca\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722821 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-tls\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.722876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722855 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.722977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.722921 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29ad770b-01c4-4963-a910-1a5010e4e79c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.723037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.723020 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebf74d8f-5bae-4c22-851c-564d30ef8088-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.725424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.725400 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.726717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.726695 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 22:01:42.726717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.726717 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 22:01:42.726717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.726700 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kgqtp\"" Oct 30 22:01:42.726956 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.726752 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebf74d8f-5bae-4c22-851c-564d30ef8088-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.728528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.728507 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 22:01:42.728528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.728517 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 22:01:42.728528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.728507 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qp8bv\"" Oct 30 22:01:42.739404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.739381 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86"] Oct 30 22:01:42.781635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.781561 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2r7\" (UniqueName: \"kubernetes.io/projected/ebf74d8f-5bae-4c22-851c-564d30ef8088-kube-api-access-6q2r7\") pod \"openshift-state-metrics-7bd7489957-xnpcn\" (UID: \"ebf74d8f-5bae-4c22-851c-564d30ef8088\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.802369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.802347 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 22:01:42.802369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.802359 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 22:01:42.823898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.823870 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-tls\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.823905 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29ad770b-01c4-4963-a910-1a5010e4e79c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.823932 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.823960 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.823984 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hbl\" (UniqueName: \"kubernetes.io/projected/6d2c6300-9238-44b0-910b-ce103e181d4a-kube-api-access-f4hbl\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824016 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-sys\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824042 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-textfile\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824070 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824096 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-root\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824122 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824158 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824184 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-wtmp\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824210 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lr2m\" (UniqueName: \"kubernetes.io/projected/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-api-access-8lr2m\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824241 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824278 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-metrics-client-ca\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824294 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29ad770b-01c4-4963-a910-1a5010e4e79c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.824705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824369 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-root\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.824976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.824957 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-wtmp\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.825092 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825075 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.825505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.825540 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825486 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.825584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825534 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d2c6300-9238-44b0-910b-ce103e181d4a-sys\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.825784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825764 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-textfile\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.825964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.825946 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d2c6300-9238-44b0-910b-ce103e181d4a-metrics-client-ca\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.827004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.826963 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-tls\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.827146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.827125 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.827719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.827698 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.828477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.828456 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d2c6300-9238-44b0-910b-ce103e181d4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.845124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.845105 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" Oct 30 22:01:42.885405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.884357 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hbl\" (UniqueName: \"kubernetes.io/projected/6d2c6300-9238-44b0-910b-ce103e181d4a-kube-api-access-f4hbl\") pod \"node-exporter-cg66d\" (UID: \"6d2c6300-9238-44b0-910b-ce103e181d4a\") " pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:42.886075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.886023 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lr2m\" (UniqueName: \"kubernetes.io/projected/29ad770b-01c4-4963-a910-1a5010e4e79c-kube-api-access-8lr2m\") pod \"kube-state-metrics-5dc8d7d89b-w7l86\" (UID: \"29ad770b-01c4-4963-a910-1a5010e4e79c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.991660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.991621 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" Oct 30 22:01:42.997230 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:42.997199 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cg66d" Oct 30 22:01:43.008314 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:43.008286 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2c6300_9238_44b0_910b_ce103e181d4a.slice/crio-015b307c8709a5ed18b7602a60864b230d87e2d9a43e0414d990d9f648d7587c WatchSource:0}: Error finding container 015b307c8709a5ed18b7602a60864b230d87e2d9a43e0414d990d9f648d7587c: Status 404 returned error can't find the container with id 015b307c8709a5ed18b7602a60864b230d87e2d9a43e0414d990d9f648d7587c Oct 30 22:01:43.267595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.267561 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn"] Oct 30 22:01:43.274509 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:43.274481 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf74d8f_5bae_4c22_851c_564d30ef8088.slice/crio-16d0f29d326285626c827cfc439c5ffc3aa8bfc4b995db8b7aa5e8f3c34190e6 WatchSource:0}: Error finding container 16d0f29d326285626c827cfc439c5ffc3aa8bfc4b995db8b7aa5e8f3c34190e6: Status 404 returned error can't find the container with id 16d0f29d326285626c827cfc439c5ffc3aa8bfc4b995db8b7aa5e8f3c34190e6 Oct 30 22:01:43.361104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.361075 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86"] Oct 30 22:01:43.371415 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:43.371380 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ad770b_01c4_4963_a910_1a5010e4e79c.slice/crio-39002fb39ae85dc05eb2666330c78a3fc065f6afa5bb926d5eb962b378c15204 WatchSource:0}: Error finding container 39002fb39ae85dc05eb2666330c78a3fc065f6afa5bb926d5eb962b378c15204: Status 404 returned error can't find the container with id 39002fb39ae85dc05eb2666330c78a3fc065f6afa5bb926d5eb962b378c15204 Oct 30 22:01:43.659036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.658958 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cg66d" event={"ID":"6d2c6300-9238-44b0-910b-ce103e181d4a","Type":"ContainerStarted","Data":"015b307c8709a5ed18b7602a60864b230d87e2d9a43e0414d990d9f648d7587c"} Oct 30 22:01:43.661164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.661134 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" event={"ID":"ebf74d8f-5bae-4c22-851c-564d30ef8088","Type":"ContainerStarted","Data":"2f1ff39af3a669dc68a13288c9054dbee0bcf2b1578b91623e0bbad942aa79ca"} Oct 30 22:01:43.661297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.661186 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" event={"ID":"ebf74d8f-5bae-4c22-851c-564d30ef8088","Type":"ContainerStarted","Data":"1671b2b6a31cc8531f2a7faf9763b9dd1c209121f62957f26451616dfea8f9e5"} Oct 30 22:01:43.661297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.661214 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" event={"ID":"ebf74d8f-5bae-4c22-851c-564d30ef8088","Type":"ContainerStarted","Data":"16d0f29d326285626c827cfc439c5ffc3aa8bfc4b995db8b7aa5e8f3c34190e6"} Oct 30 22:01:43.662457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:43.662416 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" event={"ID":"29ad770b-01c4-4963-a910-1a5010e4e79c","Type":"ContainerStarted","Data":"39002fb39ae85dc05eb2666330c78a3fc065f6afa5bb926d5eb962b378c15204"} Oct 30 22:01:45.671648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:45.671571 4305 generic.go:356] "Generic (PLEG): container finished" podID="6d2c6300-9238-44b0-910b-ce103e181d4a" containerID="a5cee5753d6b7bc1dad44a3e63c520658cea8fb690951ffaebf792164b8e1e9f" exitCode=0 Oct 30 22:01:45.671648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:45.671628 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cg66d" event={"ID":"6d2c6300-9238-44b0-910b-ce103e181d4a","Type":"ContainerDied","Data":"a5cee5753d6b7bc1dad44a3e63c520658cea8fb690951ffaebf792164b8e1e9f"} Oct 30 22:01:46.108550 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:01:46.108514 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:01:46.678970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.678932 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" event={"ID":"29ad770b-01c4-4963-a910-1a5010e4e79c","Type":"ContainerStarted","Data":"2a6c721da0d8f5f7bc2da2130b1a1052d365816bcc1be02778e1e82a68f55437"} Oct 30 22:01:46.679542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.679037 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" event={"ID":"29ad770b-01c4-4963-a910-1a5010e4e79c","Type":"ContainerStarted","Data":"c4e5661d63f6b423f4af59c8fd2c84082c14ae26bb773a6f2ed40bd2b143a5e0"} Oct 30 22:01:46.679542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.679068 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" event={"ID":"29ad770b-01c4-4963-a910-1a5010e4e79c","Type":"ContainerStarted","Data":"170b46edd44c50826cdf7705fde59c573dfcaaa4ed9de07716cd7415c4b24691"} Oct 30 22:01:46.681871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.681834 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cg66d" event={"ID":"6d2c6300-9238-44b0-910b-ce103e181d4a","Type":"ContainerStarted","Data":"4488d0e21b2cb24ee9eb65c7f92d31de80de12ad8503a567522fb49e2e7a904f"} Oct 30 22:01:46.681972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.681897 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cg66d" event={"ID":"6d2c6300-9238-44b0-910b-ce103e181d4a","Type":"ContainerStarted","Data":"e56cc5fb14dc98c125473fe7f1623a7662582c5e1173984f08d1c1b6d2c81138"} Oct 30 22:01:46.684507 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.684482 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" event={"ID":"ebf74d8f-5bae-4c22-851c-564d30ef8088","Type":"ContainerStarted","Data":"7253eea0f58ab59d0a1f9f756e4882394eae5327faa9217399f4dab260686e6a"} Oct 30 22:01:46.686304 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.686266 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mjq4c" event={"ID":"b488d9e4-12ad-4df4-9cda-25839d4620f4","Type":"ContainerStarted","Data":"85508bc65dd3de8dc6e401bb5eeb9d3db2108e871f9691b54117dbdf4ad4d5e3"} Oct 30 22:01:46.686552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.686511 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:46.706163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.706137 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mjq4c" Oct 30 22:01:46.792473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.792417 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w7l86" podStartSLOduration=2.243426863 podStartE2EDuration="4.792403924s" totalImagesPullingTime="2.548977061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.791186803 +0000 UTC m=+27.386275110" watchObservedRunningTime="2026-10-30 22:01:46.792403924 +0000 UTC m=+27.387492224" Oct 30 22:01:46.903016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:46.902960 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xnpcn" podStartSLOduration=2.38289544 podStartE2EDuration="4.902941542s" totalImagesPullingTime="2.520046102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.902083058 +0000 UTC m=+27.497171365" watchObservedRunningTime="2026-10-30 22:01:46.902941542 +0000 UTC m=+27.498029848" Oct 30 22:01:47.013829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.013751 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mjq4c" podStartSLOduration=2.679370672 podStartE2EDuration="21.013733918s" totalImagesPullingTime="18.334363246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:47.013465949 +0000 UTC m=+27.608554256" watchObservedRunningTime="2026-10-30 22:01:47.013733918 +0000 UTC m=+27.608822224" Oct 30 22:01:47.114356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.114322 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:47.114521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.114393 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:47.120006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.119983 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:47.135212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.135167 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cg66d" podStartSLOduration=2.835838785 podStartE2EDuration="5.135151754s" totalImagesPullingTime="2.213687957s" totalInitContainerRuntime="85.625012ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:47.134335285 +0000 UTC m=+27.729423610" watchObservedRunningTime="2026-10-30 22:01:47.135151754 +0000 UTC m=+27.730240273" Oct 30 22:01:47.399897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.399816 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p"] Oct 30 22:01:47.427301 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.427271 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.464109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.464079 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p"] Oct 30 22:01:47.465346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.465317 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst8d\" (UniqueName: \"kubernetes.io/projected/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-kube-api-access-dst8d\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.465503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.465368 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.476902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.476870 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 22:01:47.476902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.476870 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qpc6n\"" Oct 30 22:01:47.566161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.566122 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst8d\" (UniqueName: \"kubernetes.io/projected/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-kube-api-access-dst8d\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.566341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.566169 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.569006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.568973 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.624777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.624717 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst8d\" (UniqueName: \"kubernetes.io/projected/1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d-kube-api-access-dst8d\") pod \"monitoring-plugin-77f4f76594-l7s7p\" (UID: \"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:47.695601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.695572 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:01:47.761544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:47.761506 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:48.168682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.168602 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p"] Oct 30 22:01:48.175967 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:48.175931 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca3d1a3_da6b_4d82_9f8f_eb5da422a26d.slice/crio-5301e4a656951c99af31e2190b94d63dee2d63141787d2a424769716eb8e764c WatchSource:0}: Error finding container 5301e4a656951c99af31e2190b94d63dee2d63141787d2a424769716eb8e764c: Status 404 returned error can't find the container with id 5301e4a656951c99af31e2190b94d63dee2d63141787d2a424769716eb8e764c Oct 30 22:01:48.233542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.233516 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857zd" Oct 30 22:01:48.451704 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.451659 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:01:48.475533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.475500 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.510220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.510192 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-grfcj" Oct 30 22:01:48.516094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.516066 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:01:48.577162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577130 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577167 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577203 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps94n\" (UniqueName: \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577229 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577320 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577439 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577460 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.577584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.577484 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.578890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.578869 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 22:01:48.678782 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678750 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.678961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678791 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.678961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678827 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.678961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678855 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.678961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678877 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.678961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.678911 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps94n\" (UniqueName: \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.679269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.679092 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.679269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.679143 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.679728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.679705 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.679854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.679741 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.680398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.680376 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.680473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.680453 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.680730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.680709 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.683046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.683020 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.683140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.683027 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.696180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.696149 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" event={"ID":"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d","Type":"ContainerStarted","Data":"5301e4a656951c99af31e2190b94d63dee2d63141787d2a424769716eb8e764c"} Oct 30 22:01:48.738095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.738042 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps94n\" (UniqueName: \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\") pod \"console-744fc6d5d7-ghptk\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:48.800011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:48.799981 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:49.208639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:49.208600 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:01:49.216117 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:49.216077 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea59c54f_9a8d_475b_8bdf_91027bafcf12.slice/crio-70d3fe14f521f03a0b6d995a7a59229ed701cd0096f383de07a8bbba01b44249 WatchSource:0}: Error finding container 70d3fe14f521f03a0b6d995a7a59229ed701cd0096f383de07a8bbba01b44249: Status 404 returned error can't find the container with id 70d3fe14f521f03a0b6d995a7a59229ed701cd0096f383de07a8bbba01b44249 Oct 30 22:01:49.703022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:49.702981 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744fc6d5d7-ghptk" event={"ID":"ea59c54f-9a8d-475b-8bdf-91027bafcf12","Type":"ContainerStarted","Data":"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b"} Oct 30 22:01:49.703913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:49.703077 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744fc6d5d7-ghptk" event={"ID":"ea59c54f-9a8d-475b-8bdf-91027bafcf12","Type":"ContainerStarted","Data":"70d3fe14f521f03a0b6d995a7a59229ed701cd0096f383de07a8bbba01b44249"} Oct 30 22:01:49.816094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:49.816033 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-744fc6d5d7-ghptk" podStartSLOduration=1.8160180000000001 podStartE2EDuration="1.816018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:49.815854863 +0000 UTC m=+30.410943168" watchObservedRunningTime="2026-10-30 22:01:49.816018 +0000 UTC m=+30.411106306" Oct 30 22:01:51.714917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:51.714875 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" event={"ID":"1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d","Type":"ContainerStarted","Data":"98920fd9e3896163ca6d7621232210a0a7c2ef224aa1f9f1233bd5cfd6f00f58"} Oct 30 22:01:51.715861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:51.715834 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:51.721314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:51.721293 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" Oct 30 22:01:51.942513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:51.942451 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-l7s7p" podStartSLOduration=2.390958838 podStartE2EDuration="4.942431795s" totalImagesPullingTime="2.551472957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:51.830941837 +0000 UTC m=+32.426030144" watchObservedRunningTime="2026-10-30 22:01:51.942431795 +0000 UTC m=+32.537520100" Oct 30 22:01:58.263920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.263885 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:01:58.310880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.310573 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:01:58.372631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.372600 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.389977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.389952 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:01:58.464519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464486 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464518 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62rh\" (UniqueName: \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464541 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464686 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464733 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464837 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464866 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.464926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.464898 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.565788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.565708 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.566739 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.566717 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567429 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567602 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567677 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567695 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62rh\" (UniqueName: \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567718 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567808 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.567935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.566648 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.568224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.568209 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.568384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.567374 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.568630 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.568608 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.570479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.570459 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.570598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.570581 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.628874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.628851 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62rh\" (UniqueName: \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\") pod \"console-79f9c7488b-8xldh\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.696054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.696018 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:01:58.801116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:58.801085 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:01:59.094279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:59.094248 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:01:59.101746 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:01:59.101722 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c9dcff_1074_4be4_a14e_261ccb7f64b1.slice/crio-6f417451289b0ad55cc3d1d0b08ee92e0c0377669dc42828fdfdd3ab4d67f169 WatchSource:0}: Error finding container 6f417451289b0ad55cc3d1d0b08ee92e0c0377669dc42828fdfdd3ab4d67f169: Status 404 returned error can't find the container with id 6f417451289b0ad55cc3d1d0b08ee92e0c0377669dc42828fdfdd3ab4d67f169 Oct 30 22:01:59.744171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:59.744137 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f9c7488b-8xldh" event={"ID":"92c9dcff-1074-4be4-a14e-261ccb7f64b1","Type":"ContainerStarted","Data":"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6"} Oct 30 22:01:59.744741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:59.744196 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f9c7488b-8xldh" event={"ID":"92c9dcff-1074-4be4-a14e-261ccb7f64b1","Type":"ContainerStarted","Data":"6f417451289b0ad55cc3d1d0b08ee92e0c0377669dc42828fdfdd3ab4d67f169"} Oct 30 22:01:59.856102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:01:59.856055 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79f9c7488b-8xldh" podStartSLOduration=1.8560431259999999 podStartE2EDuration="1.856043126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:59.855404363 +0000 UTC m=+40.450492668" watchObservedRunningTime="2026-10-30 22:01:59.856043126 +0000 UTC m=+40.451131431" Oct 30 22:02:08.696776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:08.696697 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:02:08.697259 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:08.696787 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:02:08.701666 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:08.701635 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:02:08.782287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:08.782261 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:02:08.965895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:08.965824 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:02:16.107867 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:02:16.107830 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:02:23.288476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.288414 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-744fc6d5d7-ghptk" podUID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" containerName="console" containerID="cri-o://11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b" gracePeriod=15 Oct 30 22:02:23.582847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.582820 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:02:23.760238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760206 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760251 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760289 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760310 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\" (UniqueName: \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760332 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760354 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760372 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760699 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760409 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\") pod \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\" (UID: \"ea59c54f-9a8d-475b-8bdf-91027bafcf12\") " Oct 30 22:02:23.760699 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:23.760829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760695 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:23.760829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760704 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:23.760939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.760895 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:23.761187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.761157 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:23.762788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.762760 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "kube-api-access-ps94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:23.762894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.762839 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:23.762894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.762867 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config" pod "ea59c54f-9a8d-475b-8bdf-91027bafcf12" (UID: "ea59c54f-9a8d-475b-8bdf-91027bafcf12"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:23.827705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.827653 4305 generic.go:356] "Generic (PLEG): container finished" podID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" containerID="11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b" exitCode=0 Oct 30 22:02:23.827705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.827699 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744fc6d5d7-ghptk" Oct 30 22:02:23.827884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.827732 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744fc6d5d7-ghptk" event={"ID":"ea59c54f-9a8d-475b-8bdf-91027bafcf12","Type":"ContainerDied","Data":"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b"} Oct 30 22:02:23.827884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.827771 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744fc6d5d7-ghptk" event={"ID":"ea59c54f-9a8d-475b-8bdf-91027bafcf12","Type":"ContainerDied","Data":"70d3fe14f521f03a0b6d995a7a59229ed701cd0096f383de07a8bbba01b44249"} Oct 30 22:02:23.827884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.827789 4305 scope.go:122] "RemoveContainer" containerID="11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b" Oct 30 22:02:23.835934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.835919 4305 scope.go:122] "RemoveContainer" containerID="11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b" Oct 30 22:02:23.836210 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:02:23.836191 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b\": container with ID starting with 11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b not found: ID does not exist" containerID="11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b" Oct 30 22:02:23.836290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.836222 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b"} err="failed to get container status \"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b\": rpc error: code = NotFound desc = could not find container \"11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b\": container with ID starting with 11b5ce935c65b972dd232fd4f4c9f3e27daa930b0e052c978137a200d9fbfc5b not found: ID does not exist" Oct 30 22:02:23.861571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861549 4305 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861568 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps94n\" (UniqueName: \"kubernetes.io/projected/ea59c54f-9a8d-475b-8bdf-91027bafcf12-kube-api-access-ps94n\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861579 4305 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-oauth-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861588 4305 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-trusted-ca-bundle\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861597 4305 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861606 4305 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea59c54f-9a8d-475b-8bdf-91027bafcf12-console-oauth-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861614 4305 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea59c54f-9a8d-475b-8bdf-91027bafcf12-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.861685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.861622 4305 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea59c54f-9a8d-475b-8bdf-91027bafcf12-tmp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:23.980148 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.980118 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:02:23.988549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:23.988526 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-744fc6d5d7-ghptk"] Oct 30 22:02:25.516115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:25.516083 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" path="/var/lib/kubelet/pods/ea59c54f-9a8d-475b-8bdf-91027bafcf12/volumes" Oct 30 22:02:33.985841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:33.985769 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66bd6b67db-9bzrk" podUID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" containerName="console" containerID="cri-o://ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479" gracePeriod=15 Oct 30 22:02:34.224341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.224319 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:02:34.332191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332137 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\" (UniqueName: \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332168 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332201 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332226 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332263 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332291 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332309 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\") pod \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\" (UID: \"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e\") " Oct 30 22:02:34.332650 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:34.332774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332691 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.332774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332694 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.332774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.332721 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:34.334313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.334289 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "kube-api-access-jl2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:34.334411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.334313 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:34.334411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.334345 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert" pod "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" (UID: "0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:34.433388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433361 4305 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-oauth-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433389 4305 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433403 4305 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433415 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2t9\" (UniqueName: \"kubernetes.io/projected/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-kube-api-access-jl2t9\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433430 4305 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-console-oauth-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433443 4305 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-tmp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.433539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.433462 4305 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:34.866754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.866716 4305 generic.go:356] "Generic (PLEG): container finished" podID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" containerID="ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479" exitCode=0 Oct 30 22:02:34.866939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.866759 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bd6b67db-9bzrk" event={"ID":"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e","Type":"ContainerDied","Data":"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479"} Oct 30 22:02:34.866939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.866778 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66bd6b67db-9bzrk" Oct 30 22:02:34.866939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.866852 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66bd6b67db-9bzrk" event={"ID":"0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e","Type":"ContainerDied","Data":"16ce51e222e6b1f46a16020e1717550a62ed8ecd61af90a5961c84d14c20b94e"} Oct 30 22:02:34.866939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.866880 4305 scope.go:122] "RemoveContainer" containerID="ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479" Oct 30 22:02:34.875845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.875828 4305 scope.go:122] "RemoveContainer" containerID="ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479" Oct 30 22:02:34.876158 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:02:34.876138 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479\": container with ID starting with ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479 not found: ID does not exist" containerID="ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479" Oct 30 22:02:34.876217 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:34.876171 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479"} err="failed to get container status \"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479\": rpc error: code = NotFound desc = could not find container \"ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479\": container with ID starting with ef41dfb68ba340ecc7e8bd39d8ebe0d6704801da3ddaef1d5a497a64b9c86479 not found: ID does not exist" Oct 30 22:02:35.025733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:35.025707 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:02:35.031057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:35.031036 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66bd6b67db-9bzrk"] Oct 30 22:02:35.515898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:35.515868 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" path="/var/lib/kubelet/pods/0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e/volumes" Oct 30 22:02:46.108321 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:02:46.108282 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:02:52.311387 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311356 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b664cc7f-lhqws"] Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311720 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" containerName="console" Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311734 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" containerName="console" Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311743 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" containerName="console" Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311749 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" containerName="console" Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311791 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea59c54f-9a8d-475b-8bdf-91027bafcf12" containerName="console" Oct 30 22:02:52.311853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.311811 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f591f45-fb13-4a4e-b5dd-3e7eb63e7a3e" containerName="console" Oct 30 22:02:52.314212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.314193 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354293 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-console-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354328 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-oauth-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354346 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-trusted-ca-bundle\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354390 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/441f7b43-4e3e-484f-96f4-648e46cc0276-tmp\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354422 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbx6\" (UniqueName: \"kubernetes.io/projected/441f7b43-4e3e-484f-96f4-648e46cc0276-kube-api-access-rrbx6\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354462 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-oauth-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354494 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-service-ca\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.354521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.354513 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.418902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.418878 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b664cc7f-lhqws"] Oct 30 22:02:52.455657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455633 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/441f7b43-4e3e-484f-96f4-648e46cc0276-tmp\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.455778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455662 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbx6\" (UniqueName: \"kubernetes.io/projected/441f7b43-4e3e-484f-96f4-648e46cc0276-kube-api-access-rrbx6\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.455778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455683 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-oauth-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.455867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455838 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-service-ca\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.455904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455890 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.455967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.455950 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-console-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456002 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-oauth-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456071 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456021 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-trusted-ca-bundle\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456083 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/441f7b43-4e3e-484f-96f4-648e46cc0276-tmp\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456588 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-service-ca\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456685 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-console-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.456840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.456754 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-oauth-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.457044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.457025 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/441f7b43-4e3e-484f-96f4-648e46cc0276-trusted-ca-bundle\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.458101 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.458083 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-oauth-config\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.458208 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/441f7b43-4e3e-484f-96f4-648e46cc0276-console-serving-cert\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.513917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.513892 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbx6\" (UniqueName: \"kubernetes.io/projected/441f7b43-4e3e-484f-96f4-648e46cc0276-kube-api-access-rrbx6\") pod \"console-58b664cc7f-lhqws\" (UID: \"441f7b43-4e3e-484f-96f4-648e46cc0276\") " pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.623672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.623589 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:02:52.989968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:52.989943 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b664cc7f-lhqws"] Oct 30 22:02:52.997666 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:02:52.997638 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441f7b43_4e3e_484f_96f4_648e46cc0276.slice/crio-793eb3dc98276dc65cc29c4a0159ec13614163eaf1d0909bff48ed7f17006940 WatchSource:0}: Error finding container 793eb3dc98276dc65cc29c4a0159ec13614163eaf1d0909bff48ed7f17006940: Status 404 returned error can't find the container with id 793eb3dc98276dc65cc29c4a0159ec13614163eaf1d0909bff48ed7f17006940 Oct 30 22:02:53.930497 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:53.930455 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b664cc7f-lhqws" event={"ID":"441f7b43-4e3e-484f-96f4-648e46cc0276","Type":"ContainerStarted","Data":"ad0a87d964951179ce6a0971486eddbdee2925edaba68ea3d71a537394574ec4"} Oct 30 22:02:53.930996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:53.930511 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b664cc7f-lhqws" event={"ID":"441f7b43-4e3e-484f-96f4-648e46cc0276","Type":"ContainerStarted","Data":"793eb3dc98276dc65cc29c4a0159ec13614163eaf1d0909bff48ed7f17006940"} Oct 30 22:02:54.042236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:02:54.042193 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58b664cc7f-lhqws" podStartSLOduration=2.042180543 podStartE2EDuration="2.042180543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:54.041880167 +0000 UTC m=+94.636968485" watchObservedRunningTime="2026-10-30 22:02:54.042180543 +0000 UTC m=+94.637268848" Oct 30 22:03:02.623833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:02.623790 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:03:02.624285 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:02.623849 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:03:02.628487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:02.628467 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:03:02.965001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:02.964975 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b664cc7f-lhqws" Oct 30 22:03:03.107829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:03.107780 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:03:16.107747 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:03:16.107709 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:03:28.127178 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.127120 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79f9c7488b-8xldh" podUID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" containerName="console" containerID="cri-o://1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6" gracePeriod=15 Oct 30 22:03:28.371626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.371595 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:03:28.425935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.425858 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\" (UniqueName: \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.425935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.425906 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.425935 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.425993 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426036 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426073 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426098 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426159 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\") pod \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\" (UID: \"92c9dcff-1074-4be4-a14e-261ccb7f64b1\") " Oct 30 22:03:28.426468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426233 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:28.426468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426347 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:28.426569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426520 4305 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.426569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426542 4305 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92c9dcff-1074-4be4-a14e-261ccb7f64b1-tmp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.426569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426516 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:28.426569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426532 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:28.426743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.426543 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:28.428307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.428280 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:28.428424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.428310 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "kube-api-access-h62rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:28.428424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.428334 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config" pod "92c9dcff-1074-4be4-a14e-261ccb7f64b1" (UID: "92c9dcff-1074-4be4-a14e-261ccb7f64b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:28.527386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527355 4305 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-oauth-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.527386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527380 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62rh\" (UniqueName: \"kubernetes.io/projected/92c9dcff-1074-4be4-a14e-261ccb7f64b1-kube-api-access-h62rh\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.527386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527391 4305 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-trusted-ca-bundle\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.527623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527401 4305 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-oauth-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.527623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527410 4305 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-serving-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:28.527623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:28.527418 4305 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92c9dcff-1074-4be4-a14e-261ccb7f64b1-console-config\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:29.048711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.048674 4305 generic.go:356] "Generic (PLEG): container finished" podID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" containerID="1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6" exitCode=0 Oct 30 22:03:29.048924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.048721 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f9c7488b-8xldh" Oct 30 22:03:29.048924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.048739 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f9c7488b-8xldh" event={"ID":"92c9dcff-1074-4be4-a14e-261ccb7f64b1","Type":"ContainerDied","Data":"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6"} Oct 30 22:03:29.048924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.048788 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f9c7488b-8xldh" event={"ID":"92c9dcff-1074-4be4-a14e-261ccb7f64b1","Type":"ContainerDied","Data":"6f417451289b0ad55cc3d1d0b08ee92e0c0377669dc42828fdfdd3ab4d67f169"} Oct 30 22:03:29.048924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.048830 4305 scope.go:122] "RemoveContainer" containerID="1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6" Oct 30 22:03:29.056657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.056638 4305 scope.go:122] "RemoveContainer" containerID="1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6" Oct 30 22:03:29.056945 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:03:29.056928 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6\": container with ID starting with 1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6 not found: ID does not exist" containerID="1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6" Oct 30 22:03:29.056996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.056956 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6"} err="failed to get container status \"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6\": rpc error: code = NotFound desc = could not find container \"1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6\": container with ID starting with 1f2910318228862efd3a19b7fa0711941292554b363da19385de429cbd4cced6 not found: ID does not exist" Oct 30 22:03:29.214182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.214149 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:03:29.260451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.260415 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f9c7488b-8xldh"] Oct 30 22:03:29.519836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:03:29.519786 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" path="/var/lib/kubelet/pods/92c9dcff-1074-4be4-a14e-261ccb7f64b1/volumes" Oct 30 22:03:46.107690 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:03:46.107608 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:04:16.108293 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:04:16.108254 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:04:46.107687 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:04:46.107650 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:05:16.108665 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:05:16.108625 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:05:28.245566 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.245529 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d"] Oct 30 22:05:28.245993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.245814 4305 cpu_manager.go:414] "containerMap: removing container" podUID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" containerName="console" Oct 30 22:05:28.245993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.245829 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" containerName="console" Oct 30 22:05:28.245993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.245885 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c9dcff-1074-4be4-a14e-261ccb7f64b1" containerName="console" Oct 30 22:05:28.247825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.247792 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.345826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.345764 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 22:05:28.346118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.346016 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7pwvt\"" Oct 30 22:05:28.346256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.346167 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 22:05:28.347109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.347083 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d"] Oct 30 22:05:28.368788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.368752 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.368975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.368809 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7l5\" (UniqueName: \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.368975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.368901 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.469658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.469612 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.469843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.469673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7l5\" (UniqueName: \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.469843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.469703 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.470186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.470162 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.470292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.470192 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.507355 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.507287 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:05:28.509228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.509212 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.539196 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.539173 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7l5\" (UniqueName: \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.563772 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.563751 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 22:05:28.564848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.564831 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:28.574556 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.574538 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 22:05:28.578112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.577974 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:05:28.578112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.578024 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:05:28.578112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.578024 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 22:05:28.578112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.578074 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:05:28.578112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.578091 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7vbmp\"" Oct 30 22:05:28.581822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.581778 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:05:28.671022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.670991 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671030 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671060 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fx2\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671173 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671224 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671275 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.671505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.671478 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772446 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772481 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772500 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772622 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fx2\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772678 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772709 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.772765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.772744 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.773289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.773261 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.774859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.774828 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.774859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.774854 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.775062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.774904 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.775062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.774915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.830213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.830189 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.830420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.830401 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fx2\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\") pod \"istiod-openshift-gateway-7876bfcb74-f24xf\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:28.931051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:28.930950 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d"] Oct 30 22:05:28.939338 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:05:28.939313 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0306d160_f5bf_4f4a_96e0_27fa5c793468.slice/crio-e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290 WatchSource:0}: Error finding container e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290: Status 404 returned error can't find the container with id e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290 Oct 30 22:05:29.119840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:29.119752 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:29.439534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:29.439500 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerStarted","Data":"e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290"} Oct 30 22:05:29.536561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:29.536532 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:05:29.543891 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:05:29.543865 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61f2bc4_a638_40c7_bbfe_49049b758602.slice/crio-903d552ed313633b19dcc8b2addadbadbcb925edaa08d3b285b291f8c82d41ae WatchSource:0}: Error finding container 903d552ed313633b19dcc8b2addadbadbcb925edaa08d3b285b291f8c82d41ae: Status 404 returned error can't find the container with id 903d552ed313633b19dcc8b2addadbadbcb925edaa08d3b285b291f8c82d41ae Oct 30 22:05:30.445652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:30.445613 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" event={"ID":"e61f2bc4-a638-40c7-bbfe-49049b758602","Type":"ContainerStarted","Data":"903d552ed313633b19dcc8b2addadbadbcb925edaa08d3b285b291f8c82d41ae"} Oct 30 22:05:33.661588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:33.661552 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:05:33.662023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:33.661632 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:05:34.464115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:34.464081 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" event={"ID":"e61f2bc4-a638-40c7-bbfe-49049b758602","Type":"ContainerStarted","Data":"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f"} Oct 30 22:05:34.464538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:34.464520 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:34.466191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:34.466151 4305 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-f24xf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:34.466292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:34.466204 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:34.658045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:34.657988 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" podStartSLOduration=2.543102943 podStartE2EDuration="6.657969136s" totalImagesPullingTime="4.114866193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:34.623260839 +0000 UTC m=+255.218349147" watchObservedRunningTime="2026-10-30 22:05:34.657969136 +0000 UTC m=+255.253057443" Oct 30 22:05:35.468338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:35.468296 4305 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-f24xf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:35.468847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:35.468360 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:36.472762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:36.472723 4305 generic.go:356] "Generic (PLEG): container finished" podID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerID="8d0561e41eba08c5bfab14668efb1ee76919684161fbb9fcfa6491f566d180a3" exitCode=0 Oct 30 22:05:36.473207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:36.472829 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerDied","Data":"8d0561e41eba08c5bfab14668efb1ee76919684161fbb9fcfa6491f566d180a3"} Oct 30 22:05:38.468874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:38.468842 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:05:38.480899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:38.480870 4305 generic.go:356] "Generic (PLEG): container finished" podID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerID="791c90954d724195fb8478e31a28a13da0d3ac1fd44ad0d65de3636f71ab089a" exitCode=0 Oct 30 22:05:38.481065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:38.480927 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerDied","Data":"791c90954d724195fb8478e31a28a13da0d3ac1fd44ad0d65de3636f71ab089a"} Oct 30 22:05:45.508641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:45.508603 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerStarted","Data":"aa1d9cbd3d41bf8ef940f45f93b170b3479220cc1f8164cdceab3ab6f561bccf"} Oct 30 22:05:45.620365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:45.620302 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" podStartSLOduration=2.799896016 podStartE2EDuration="17.620289959s" totalImagesPullingTime="14.820393943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:45.619360988 +0000 UTC m=+266.214449293" watchObservedRunningTime="2026-10-30 22:05:45.620289959 +0000 UTC m=+266.215378265" Oct 30 22:05:46.108238 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:05:46.108156 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:05:46.514049 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:46.514008 4305 generic.go:356] "Generic (PLEG): container finished" podID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerID="aa1d9cbd3d41bf8ef940f45f93b170b3479220cc1f8164cdceab3ab6f561bccf" exitCode=0 Oct 30 22:05:46.514454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:46.514067 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerDied","Data":"aa1d9cbd3d41bf8ef940f45f93b170b3479220cc1f8164cdceab3ab6f561bccf"} Oct 30 22:05:47.636741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.636715 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:47.730027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.729997 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\") pod \"0306d160-f5bf-4f4a-96e0-27fa5c793468\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " Oct 30 22:05:47.730220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.730047 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\") pod \"0306d160-f5bf-4f4a-96e0-27fa5c793468\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " Oct 30 22:05:47.730220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.730092 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\" (UniqueName: \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\") pod \"0306d160-f5bf-4f4a-96e0-27fa5c793468\" (UID: \"0306d160-f5bf-4f4a-96e0-27fa5c793468\") " Oct 30 22:05:47.730683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.730655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle" pod "0306d160-f5bf-4f4a-96e0-27fa5c793468" (UID: "0306d160-f5bf-4f4a-96e0-27fa5c793468"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:47.732307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.732286 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5" pod "0306d160-f5bf-4f4a-96e0-27fa5c793468" (UID: "0306d160-f5bf-4f4a-96e0-27fa5c793468"). InnerVolumeSpecName "kube-api-access-qp7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:47.735930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.735900 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util" pod "0306d160-f5bf-4f4a-96e0-27fa5c793468" (UID: "0306d160-f5bf-4f4a-96e0-27fa5c793468"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:47.830866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.830759 4305 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-bundle\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:47.830866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.830789 4305 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0306d160-f5bf-4f4a-96e0-27fa5c793468-util\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:47.830866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:47.830824 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7l5\" (UniqueName: \"kubernetes.io/projected/0306d160-f5bf-4f4a-96e0-27fa5c793468-kube-api-access-qp7l5\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:48.521764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:48.521727 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" event={"ID":"0306d160-f5bf-4f4a-96e0-27fa5c793468","Type":"ContainerDied","Data":"e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290"} Oct 30 22:05:48.521764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:48.521748 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cpg7d" Oct 30 22:05:48.521764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:05:48.521770 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91931c7d4b4a1801bad70963d940b054a281406e75f8742d1efbaeea9c71290" Oct 30 22:06:13.621863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.621821 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c"] Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622223 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="extract" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622240 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="extract" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622251 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="pull" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622259 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="pull" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622300 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="util" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622307 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="util" Oct 30 22:06:13.622474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.622378 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0306d160-f5bf-4f4a-96e0-27fa5c793468" containerName="extract" Oct 30 22:06:13.624318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.624297 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.704878 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.704851 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 22:06:13.716652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716628 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.716731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716666 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.716731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716700 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glmj\" (UniqueName: \"kubernetes.io/projected/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-kube-api-access-7glmj\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.716847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716820 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.716892 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716854 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.716892 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.716883 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.719288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.719273 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 22:06:13.719375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.719360 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 22:06:13.719416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.719367 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8hr44\"" Oct 30 22:06:13.719517 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.719502 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 22:06:13.719586 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.719575 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 22:06:13.817385 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817354 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.817526 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817391 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.817567 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817516 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.817601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817582 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glmj\" (UniqueName: \"kubernetes.io/projected/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-kube-api-access-7glmj\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.817640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817627 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.817690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817667 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.818030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.817998 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.819785 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.819759 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.820000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.819982 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.820138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.820117 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.820188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.820172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.876660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.876592 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glmj\" (UniqueName: \"kubernetes.io/projected/cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc-kube-api-access-7glmj\") pod \"opendatahub-operator-controller-manager-66d654f8fd-sp28c\" (UID: \"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:13.901913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.901881 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c"] Oct 30 22:06:13.936902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:13.936870 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:14.422997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:14.422965 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c"] Oct 30 22:06:14.429511 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:06:14.429475 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb5bcc7_d591_48b5_a9e7_9007a5e1d9fc.slice/crio-3a09b7eff6b4470d5380630dae679ccaae035061dbddd391f6d21863dc14a1fe WatchSource:0}: Error finding container 3a09b7eff6b4470d5380630dae679ccaae035061dbddd391f6d21863dc14a1fe: Status 404 returned error can't find the container with id 3a09b7eff6b4470d5380630dae679ccaae035061dbddd391f6d21863dc14a1fe Oct 30 22:06:14.610708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:14.610675 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" event={"ID":"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc","Type":"ContainerStarted","Data":"3a09b7eff6b4470d5380630dae679ccaae035061dbddd391f6d21863dc14a1fe"} Oct 30 22:06:16.114040 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:06:16.114003 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:06:18.627600 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:18.627563 4305 generic.go:356] "Generic (PLEG): container finished" podID="cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc" containerID="477bb219e684d04ef26ba0f182af55c0e62c20769590336acb737bdf7b5a939f" exitCode=0 Oct 30 22:06:18.628007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:18.627645 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" event={"ID":"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc","Type":"ContainerDied","Data":"477bb219e684d04ef26ba0f182af55c0e62c20769590336acb737bdf7b5a939f"} Oct 30 22:06:19.534141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:19.534112 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:06:19.535228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:19.535206 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:06:19.633573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:19.633523 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" event={"ID":"cdb5bcc7-d591-48b5-a9e7-9007a5e1d9fc","Type":"ContainerStarted","Data":"20861ea71f65773f4f2bb71fba72d1b95f29faf404e0bc6eeb212bf3e306e0ea"} Oct 30 22:06:19.634091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:19.633674 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:19.862229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:19.862127 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" podStartSLOduration=1.955891737 podStartE2EDuration="6.862113166s" totalImagesPullingTime="4.446083904s" totalInitContainerRuntime="460.137525ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:19.815598912 +0000 UTC m=+300.410687230" watchObservedRunningTime="2026-10-30 22:06:19.862113166 +0000 UTC m=+300.457201468" Oct 30 22:06:30.639685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:30.639654 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-sp28c" Oct 30 22:06:44.291195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.291120 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm"] Oct 30 22:06:44.293202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.293182 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.392292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.392261 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 22:06:44.392292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.392274 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 22:06:44.392481 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.392325 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ssrkh\"" Oct 30 22:06:44.393604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.393556 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm"] Oct 30 22:06:44.463080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.463045 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54gx\" (UniqueName: \"kubernetes.io/projected/5cda4a18-49ba-4716-b38e-7aa00c726785-kube-api-access-f54gx\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.463221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.463089 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cda4a18-49ba-4716-b38e-7aa00c726785-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.463221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.463125 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cda4a18-49ba-4716-b38e-7aa00c726785-tmp\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.564655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.564573 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54gx\" (UniqueName: \"kubernetes.io/projected/5cda4a18-49ba-4716-b38e-7aa00c726785-kube-api-access-f54gx\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.564655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.564625 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cda4a18-49ba-4716-b38e-7aa00c726785-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.564938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.564673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cda4a18-49ba-4716-b38e-7aa00c726785-tmp\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.566978 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.566949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cda4a18-49ba-4716-b38e-7aa00c726785-tmp\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.567153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.567135 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cda4a18-49ba-4716-b38e-7aa00c726785-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.621766 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.621735 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54gx\" (UniqueName: \"kubernetes.io/projected/5cda4a18-49ba-4716-b38e-7aa00c726785-kube-api-access-f54gx\") pod \"kube-auth-proxy-784dfbd64d-7pfnm\" (UID: \"5cda4a18-49ba-4716-b38e-7aa00c726785\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:44.904452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:44.904369 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" Oct 30 22:06:45.274729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:45.274695 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm"] Oct 30 22:06:45.280647 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:06:45.280619 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cda4a18_49ba_4716_b38e_7aa00c726785.slice/crio-34371cdebb4e3b415f0e197688497a0b7b42189a62eb71e6968659b0d8117122 WatchSource:0}: Error finding container 34371cdebb4e3b415f0e197688497a0b7b42189a62eb71e6968659b0d8117122: Status 404 returned error can't find the container with id 34371cdebb4e3b415f0e197688497a0b7b42189a62eb71e6968659b0d8117122 Oct 30 22:06:45.282612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:45.282596 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:06:45.722210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:45.722173 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" event={"ID":"5cda4a18-49ba-4716-b38e-7aa00c726785","Type":"ContainerStarted","Data":"34371cdebb4e3b415f0e197688497a0b7b42189a62eb71e6968659b0d8117122"} Oct 30 22:06:46.108627 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:06:46.108550 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:06:48.160046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.160011 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:06:48.162633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.162610 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.211811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.211725 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 22:06:48.211980 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.211874 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 22:06:48.257663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.257642 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 22:06:48.258900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.258877 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:06:48.296910 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.296883 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grqp\" (UniqueName: \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.297011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.296967 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.303894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.303875 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bcwpx\"" Oct 30 22:06:48.397581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.397554 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.397845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.397611 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grqp\" (UniqueName: \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.398169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.398147 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.407944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.407920 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:06:48.457372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.457346 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grqp\" (UniqueName: \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\") pod \"limitador-limitador-74dfcfb894-4cgxr\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.478154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.478131 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:48.735899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.735827 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" event={"ID":"5cda4a18-49ba-4716-b38e-7aa00c726785","Type":"ContainerStarted","Data":"26f553c5f300d6329664fa7c275952559f772aa9a077d4a7872c036348b221a4"} Oct 30 22:06:48.848875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.848847 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:06:48.852121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:48.852039 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-7pfnm" podStartSLOduration=1.900924866 podStartE2EDuration="4.852020361s" totalImagesPullingTime="2.951095495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:48.851404872 +0000 UTC m=+329.446493180" watchObservedRunningTime="2026-10-30 22:06:48.852020361 +0000 UTC m=+329.447108669" Oct 30 22:06:48.856490 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:06:48.856460 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56ef5f0_5dd2_42fd_b743_a0ea582ba7d7.slice/crio-6253c5abedaf9c966ea6cc1b92bbcf7fc518dbac7d70ce50d5e0e5fd28e96ca6 WatchSource:0}: Error finding container 6253c5abedaf9c966ea6cc1b92bbcf7fc518dbac7d70ce50d5e0e5fd28e96ca6: Status 404 returned error can't find the container with id 6253c5abedaf9c966ea6cc1b92bbcf7fc518dbac7d70ce50d5e0e5fd28e96ca6 Oct 30 22:06:49.740786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:49.740753 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" event={"ID":"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7","Type":"ContainerStarted","Data":"6253c5abedaf9c966ea6cc1b92bbcf7fc518dbac7d70ce50d5e0e5fd28e96ca6"} Oct 30 22:06:53.762541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:53.762498 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" event={"ID":"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7","Type":"ContainerStarted","Data":"b9fb648ab7b0c8a68af1e43e4324413463ee028f494ab7e0d980e70285e4073e"} Oct 30 22:06:53.762951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:53.762648 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:06:53.872592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:06:53.872542 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" podStartSLOduration=1.844781558 podStartE2EDuration="5.872528457s" totalImagesPullingTime="4.027746899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.87124654 +0000 UTC m=+334.466334846" watchObservedRunningTime="2026-10-30 22:06:53.872528457 +0000 UTC m=+334.467616762" Oct 30 22:07:04.767075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:04.767046 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:07:08.357195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:08.357161 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:07:08.357603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:08.357354 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" podUID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" containerName="limitador" containerID="cri-o://b9fb648ab7b0c8a68af1e43e4324413463ee028f494ab7e0d980e70285e4073e" gracePeriod=30 Oct 30 22:07:08.820240 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:08.820212 4305 generic.go:356] "Generic (PLEG): container finished" podID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" containerID="b9fb648ab7b0c8a68af1e43e4324413463ee028f494ab7e0d980e70285e4073e" exitCode=0 Oct 30 22:07:08.820352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:08.820261 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" event={"ID":"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7","Type":"ContainerDied","Data":"b9fb648ab7b0c8a68af1e43e4324413463ee028f494ab7e0d980e70285e4073e"} Oct 30 22:07:09.288113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.288091 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:07:09.362270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.362230 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\" (UniqueName: \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\") pod \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " Oct 30 22:07:09.362691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.362299 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\" (UniqueName: \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\") pod \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\" (UID: \"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7\") " Oct 30 22:07:09.362691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.362587 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file" pod "c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" (UID: "c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:09.364393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.364369 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp" pod "c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" (UID: "c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7"). InnerVolumeSpecName "kube-api-access-7grqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:09.447280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.447254 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd"] Oct 30 22:07:09.447535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.447523 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" containerName="limitador" Oct 30 22:07:09.447580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.447537 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" containerName="limitador" Oct 30 22:07:09.447613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.447597 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" containerName="limitador" Oct 30 22:07:09.450348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.450327 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.462685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.462663 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.462808 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.462691 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea49084-0340-4d8d-86b0-fc05b559a2f7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.462808 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.462722 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsfq\" (UniqueName: \"kubernetes.io/projected/dea49084-0340-4d8d-86b0-fc05b559a2f7-kube-api-access-9qsfq\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.462938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.462872 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.463004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.462988 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grqp\" (UniqueName: \"kubernetes.io/projected/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-kube-api-access-7grqp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:09.463050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.463014 4305 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7-config-file\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:09.499629 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.499607 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-zgjmh\"" Oct 30 22:07:09.499731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.499618 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 22:07:09.499731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.499679 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 22:07:09.563629 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.563602 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.563764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.563638 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea49084-0340-4d8d-86b0-fc05b559a2f7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.563764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.563695 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsfq\" (UniqueName: \"kubernetes.io/projected/dea49084-0340-4d8d-86b0-fc05b559a2f7-kube-api-access-9qsfq\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.563764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.563734 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.563982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.563962 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.564098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.564078 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea49084-0340-4d8d-86b0-fc05b559a2f7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.592712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.592678 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 22:07:09.595757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.595581 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dea49084-0340-4d8d-86b0-fc05b559a2f7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.595757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.595642 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd"] Oct 30 22:07:09.621626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.621597 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsfq\" (UniqueName: \"kubernetes.io/projected/dea49084-0340-4d8d-86b0-fc05b559a2f7-kube-api-access-9qsfq\") pod \"kserve-module-controller-manager-7566cf5fc8-zrwhd\" (UID: \"dea49084-0340-4d8d-86b0-fc05b559a2f7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.761870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.761809 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:09.825699 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.825674 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" Oct 30 22:07:09.825818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.825673 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4cgxr" event={"ID":"c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7","Type":"ContainerDied","Data":"6253c5abedaf9c966ea6cc1b92bbcf7fc518dbac7d70ce50d5e0e5fd28e96ca6"} Oct 30 22:07:09.825873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.825830 4305 scope.go:122] "RemoveContainer" containerID="b9fb648ab7b0c8a68af1e43e4324413463ee028f494ab7e0d980e70285e4073e" Oct 30 22:07:09.978533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.978498 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:07:09.984629 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:09.984601 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4cgxr"] Oct 30 22:07:10.177598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:10.177523 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd"] Oct 30 22:07:10.184292 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:07:10.184264 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea49084_0340_4d8d_86b0_fc05b559a2f7.slice/crio-b090fb2c306dd1921223254ba95f463e556f7e257b79a9e28bdd8206bdfa526a WatchSource:0}: Error finding container b090fb2c306dd1921223254ba95f463e556f7e257b79a9e28bdd8206bdfa526a: Status 404 returned error can't find the container with id b090fb2c306dd1921223254ba95f463e556f7e257b79a9e28bdd8206bdfa526a Oct 30 22:07:10.831577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:10.831546 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" event={"ID":"dea49084-0340-4d8d-86b0-fc05b559a2f7","Type":"ContainerStarted","Data":"b090fb2c306dd1921223254ba95f463e556f7e257b79a9e28bdd8206bdfa526a"} Oct 30 22:07:11.516739 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:11.516708 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7" path="/var/lib/kubelet/pods/c56ef5f0-5dd2-42fd-b743-a0ea582ba7d7/volumes" Oct 30 22:07:12.840234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:12.840174 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" event={"ID":"dea49084-0340-4d8d-86b0-fc05b559a2f7","Type":"ContainerStarted","Data":"ae503c78893a9503ed0970aa599ae7f9c6e2a44c3dc9ae8df001b5257a837051"} Oct 30 22:07:12.840683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:12.840320 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:13.002243 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:13.002194 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" podStartSLOduration=1.774406194 podStartE2EDuration="4.002182029s" totalImagesPullingTime="2.227775835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:13.001369976 +0000 UTC m=+353.596458283" watchObservedRunningTime="2026-10-30 22:07:13.002182029 +0000 UTC m=+353.597270334" Oct 30 22:07:15.909658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:15.909626 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54"] Oct 30 22:07:15.913176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:15.913152 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:15.962497 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:15.962473 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 22:07:16.008268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.008242 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:16.008413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.008395 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:16.008460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.008436 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-49n77\"" Oct 30 22:07:16.009596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.009579 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54"] Oct 30 22:07:16.011711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.011691 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4kj\" (UniqueName: \"kubernetes.io/projected/40c9a0b6-ea50-4947-bde5-cab611ff1779-kube-api-access-2x4kj\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.011757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.011719 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c9a0b6-ea50-4947-bde5-cab611ff1779-data\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.107918 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:16.107874 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:07:16.112341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.112313 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4kj\" (UniqueName: \"kubernetes.io/projected/40c9a0b6-ea50-4947-bde5-cab611ff1779-kube-api-access-2x4kj\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.112472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.112346 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c9a0b6-ea50-4947-bde5-cab611ff1779-data\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.112751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.112731 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40c9a0b6-ea50-4947-bde5-cab611ff1779-data\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.171549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.171482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4kj\" (UniqueName: \"kubernetes.io/projected/40c9a0b6-ea50-4947-bde5-cab611ff1779-kube-api-access-2x4kj\") pod \"postgres-7cdbf6dc44-fjg54\" (UID: \"40c9a0b6-ea50-4947-bde5-cab611ff1779\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.225513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.225481 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:16.593518 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.593483 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54"] Oct 30 22:07:16.599621 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:07:16.599577 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c9a0b6_ea50_4947_bde5_cab611ff1779.slice/crio-4cdca7fc20276d3e8e2f15399cfbaa132c658416533ce29d0edb8375540402b8 WatchSource:0}: Error finding container 4cdca7fc20276d3e8e2f15399cfbaa132c658416533ce29d0edb8375540402b8: Status 404 returned error can't find the container with id 4cdca7fc20276d3e8e2f15399cfbaa132c658416533ce29d0edb8375540402b8 Oct 30 22:07:16.856982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:16.856885 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" event={"ID":"40c9a0b6-ea50-4947-bde5-cab611ff1779","Type":"ContainerStarted","Data":"4cdca7fc20276d3e8e2f15399cfbaa132c658416533ce29d0edb8375540402b8"} Oct 30 22:07:23.332334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:23.332307 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 22:07:23.888818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:23.888768 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" event={"ID":"40c9a0b6-ea50-4947-bde5-cab611ff1779","Type":"ContainerStarted","Data":"12b58ca4107c3fd612d4e23c81aa12728de8d4b3ac6431207e5011e8dd995e6f"} Oct 30 22:07:23.888993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:23.888908 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:23.997344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:23.997295 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" podStartSLOduration=2.314281939 podStartE2EDuration="8.997274586s" totalImagesPullingTime="6.682992647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:23.996619891 +0000 UTC m=+364.591708199" watchObservedRunningTime="2026-10-30 22:07:23.997274586 +0000 UTC m=+364.592362892" Oct 30 22:07:29.921956 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:29.921927 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fjg54" Oct 30 22:07:33.846141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:33.846109 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zrwhd" Oct 30 22:07:37.275032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.275000 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:37.277075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.277055 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.326347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.326318 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ct9hv\"" Oct 30 22:07:37.326469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.326366 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:07:37.350863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.350839 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.350997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.350898 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.351044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.351020 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4729d\" (UniqueName: \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.372438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.372412 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:07:37.372438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.372435 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:07:37.373811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.373774 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:37.452083 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.452046 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4729d\" (UniqueName: \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.452280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.452104 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.452280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.452177 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.455824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.454995 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.455824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.455640 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.511585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.511550 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4729d\" (UniqueName: \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\") pod \"maas-controller-579cf7c6d-twmz6\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.587976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.587892 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:37.726397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.726369 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:07:37.728667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.728644 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.753848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.753823 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfm5\" (UniqueName: \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.753954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.753864 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.754001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.753962 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.790861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.790831 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:07:37.854983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.854921 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfm5\" (UniqueName: \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.854983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.854957 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.854983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.854983 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.857327 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.857292 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.857440 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.857358 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.913079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.913051 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfm5\" (UniqueName: \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\") pod \"maas-controller-8cf787766-jpmcd\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:37.954588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:37.954561 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:37.960693 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:07:37.960671 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c3a128_e67b_4cf6_8328_bc07a8b9efae.slice/crio-da28a6d67ccc0323fad85b97818c7f293e67b04be0ed14b93b57b604b879d625 WatchSource:0}: Error finding container da28a6d67ccc0323fad85b97818c7f293e67b04be0ed14b93b57b604b879d625: Status 404 returned error can't find the container with id da28a6d67ccc0323fad85b97818c7f293e67b04be0ed14b93b57b604b879d625 Oct 30 22:07:38.040218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:38.040195 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:38.404648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:38.404613 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:07:38.410940 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:07:38.410908 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f3ca0f_c4e6_4d19_8e70_1f653e0b8734.slice/crio-34266fb778ec26c1498ceafbf743d4b438db75ab7d86509e8a3d0d9f797f13b1 WatchSource:0}: Error finding container 34266fb778ec26c1498ceafbf743d4b438db75ab7d86509e8a3d0d9f797f13b1: Status 404 returned error can't find the container with id 34266fb778ec26c1498ceafbf743d4b438db75ab7d86509e8a3d0d9f797f13b1 Oct 30 22:07:38.945369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:38.945330 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerStarted","Data":"34266fb778ec26c1498ceafbf743d4b438db75ab7d86509e8a3d0d9f797f13b1"} Oct 30 22:07:38.946420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:38.946399 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twmz6" event={"ID":"30c3a128-e67b-4cf6-8328-bc07a8b9efae","Type":"ContainerStarted","Data":"da28a6d67ccc0323fad85b97818c7f293e67b04be0ed14b93b57b604b879d625"} Oct 30 22:07:41.960889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:41.960841 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerStarted","Data":"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786"} Oct 30 22:07:41.961377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:41.961037 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:41.962334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:41.962310 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twmz6" event={"ID":"30c3a128-e67b-4cf6-8328-bc07a8b9efae","Type":"ContainerStarted","Data":"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2"} Oct 30 22:07:41.962474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:41.962458 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:42.073493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:42.073381 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8cf787766-jpmcd" podStartSLOduration=1.686970174 podStartE2EDuration="5.073367862s" totalImagesPullingTime="3.386397688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:42.072310616 +0000 UTC m=+382.667398923" watchObservedRunningTime="2026-10-30 22:07:42.073367862 +0000 UTC m=+382.668456197" Oct 30 22:07:42.184452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:42.184394 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-twmz6" podStartSLOduration=1.65749488 podStartE2EDuration="5.184378482s" totalImagesPullingTime="3.526883602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:42.183003272 +0000 UTC m=+382.778091589" watchObservedRunningTime="2026-10-30 22:07:42.184378482 +0000 UTC m=+382.779466788" Oct 30 22:07:44.697270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.697235 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5"] Oct 30 22:07:44.699590 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.699568 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.713687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.713657 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.713819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.713698 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bmx\" (UniqueName: \"kubernetes.io/projected/16f613be-f4da-466b-9e94-f068104933fe-kube-api-access-b5bmx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.713819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.713791 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.713955 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.713830 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.748432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.748407 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 22:07:44.748559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.748441 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-j7pb9\"" Oct 30 22:07:44.794624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.794598 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 22:07:44.794624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.794599 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 22:07:44.796830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.796790 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5"] Oct 30 22:07:44.814861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.814833 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.814989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.814866 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.814989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.814925 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.814989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.814948 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bmx\" (UniqueName: \"kubernetes.io/projected/16f613be-f4da-466b-9e94-f068104933fe-kube-api-access-b5bmx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.815143 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:44.815012 4305 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 22:07:44.815143 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:44.815037 4305 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 30 22:07:44.815143 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:44.815090 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert podName:16f613be-f4da-466b-9e94-f068104933fe nodeName:}" failed. No retries permitted until 2026-10-30 22:07:45.315070564 +0000 UTC m=+385.910158865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" (UID: "16f613be-f4da-466b-9e94-f068104933fe") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 22:07:44.815143 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:44.815109 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source podName:16f613be-f4da-466b-9e94-f068104933fe nodeName:}" failed. No retries permitted until 2026-10-30 22:07:45.315101048 +0000 UTC m=+385.910189343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" (UID: "16f613be-f4da-466b-9e94-f068104933fe") : configmap "usage-tenancy-proxy-source" not found Oct 30 22:07:44.815547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.815529 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:44.873083 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:44.873051 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bmx\" (UniqueName: \"kubernetes.io/projected/16f613be-f4da-466b-9e94-f068104933fe-kube-api-access-b5bmx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.319962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.319923 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.320132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.319983 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.320750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.320731 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/16f613be-f4da-466b-9e94-f068104933fe-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.322355 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.322333 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f613be-f4da-466b-9e94-f068104933fe-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4pmf5\" (UID: \"16f613be-f4da-466b-9e94-f068104933fe\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.611854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.611755 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:07:45.978480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:45.978445 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5"] Oct 30 22:07:45.985416 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:07:45.985392 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f613be_f4da_466b_9e94_f068104933fe.slice/crio-4264145ce0b67b6931b7545a0beb0cf7107d93d45908b38bb36b374c44977e3c WatchSource:0}: Error finding container 4264145ce0b67b6931b7545a0beb0cf7107d93d45908b38bb36b374c44977e3c: Status 404 returned error can't find the container with id 4264145ce0b67b6931b7545a0beb0cf7107d93d45908b38bb36b374c44977e3c Oct 30 22:07:46.107543 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:46.107510 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:07:46.983310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:46.983257 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" event={"ID":"16f613be-f4da-466b-9e94-f068104933fe","Type":"ContainerStarted","Data":"4264145ce0b67b6931b7545a0beb0cf7107d93d45908b38bb36b374c44977e3c"} Oct 30 22:07:52.968991 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:52.968958 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:52.969976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:52.969951 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:07:53.226910 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.226612 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:53.227226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.227192 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-twmz6" podUID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" containerName="manager" containerID="cri-o://5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2" gracePeriod=30 Oct 30 22:07:53.589660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.589620 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:53.597912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.597881 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\") pod \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " Oct 30 22:07:53.598094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.597931 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\" (UniqueName: \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\") pod \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " Oct 30 22:07:53.598094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.597985 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\") pod \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\" (UID: \"30c3a128-e67b-4cf6-8328-bc07a8b9efae\") " Oct 30 22:07:53.601784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.601754 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d" pod "30c3a128-e67b-4cf6-8328-bc07a8b9efae" (UID: "30c3a128-e67b-4cf6-8328-bc07a8b9efae"). InnerVolumeSpecName "kube-api-access-4729d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:53.602350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.602326 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs" pod "30c3a128-e67b-4cf6-8328-bc07a8b9efae" (UID: "30c3a128-e67b-4cf6-8328-bc07a8b9efae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:53.602918 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.602892 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert" pod "30c3a128-e67b-4cf6-8328-bc07a8b9efae" (UID: "30c3a128-e67b-4cf6-8328-bc07a8b9efae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:53.699062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.699022 4305 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-webhook-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:53.699062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.699056 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4729d\" (UniqueName: \"kubernetes.io/projected/30c3a128-e67b-4cf6-8328-bc07a8b9efae-kube-api-access-4729d\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:53.699266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:53.699073 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30c3a128-e67b-4cf6-8328-bc07a8b9efae-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:54.019262 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.019223 4305 generic.go:356] "Generic (PLEG): container finished" podID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" containerID="5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2" exitCode=0 Oct 30 22:07:54.019757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.019314 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twmz6" event={"ID":"30c3a128-e67b-4cf6-8328-bc07a8b9efae","Type":"ContainerDied","Data":"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2"} Oct 30 22:07:54.019757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.019360 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twmz6" event={"ID":"30c3a128-e67b-4cf6-8328-bc07a8b9efae","Type":"ContainerDied","Data":"da28a6d67ccc0323fad85b97818c7f293e67b04be0ed14b93b57b604b879d625"} Oct 30 22:07:54.019757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.019391 4305 scope.go:122] "RemoveContainer" containerID="5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2" Oct 30 22:07:54.019757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.019542 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twmz6" Oct 30 22:07:54.036423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.036405 4305 scope.go:122] "RemoveContainer" containerID="5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2" Oct 30 22:07:54.036758 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:07:54.036707 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2\": container with ID starting with 5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2 not found: ID does not exist" containerID="5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2" Oct 30 22:07:54.036758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.036749 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2"} err="failed to get container status \"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2\": rpc error: code = NotFound desc = could not find container \"5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2\": container with ID starting with 5f9622741f332fe5bd6d3107aa9d4f7ed1991426c83ea50a34fd78ebf915d8c2 not found: ID does not exist" Oct 30 22:07:54.178125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.178049 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:54.225046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:54.225011 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twmz6"] Oct 30 22:07:55.516876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:07:55.516846 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" path="/var/lib/kubelet/pods/30c3a128-e67b-4cf6-8328-bc07a8b9efae/volumes" Oct 30 22:08:00.051022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:00.050977 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" event={"ID":"16f613be-f4da-466b-9e94-f068104933fe","Type":"ContainerStarted","Data":"9cfaeca18811ca2e08a05404a9468a3d562cc78319bf69b61772cc8ae7a0292c"} Oct 30 22:08:00.051437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:00.051105 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:08:00.162790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:00.162738 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" podStartSLOduration=2.937284772 podStartE2EDuration="16.162724822s" totalImagesPullingTime="13.22544005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:00.161280679 +0000 UTC m=+400.756368989" watchObservedRunningTime="2026-10-30 22:08:00.162724822 +0000 UTC m=+400.757813128" Oct 30 22:08:11.060582 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:11.060505 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4pmf5" Oct 30 22:08:13.089237 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.089209 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7"] Oct 30 22:08:13.089607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.089527 4305 cpu_manager.go:414] "containerMap: removing container" podUID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" containerName="manager" Oct 30 22:08:13.089607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.089538 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" containerName="manager" Oct 30 22:08:13.089607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.089602 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c3a128-e67b-4cf6-8328-bc07a8b9efae" containerName="manager" Oct 30 22:08:13.094182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.094163 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.160981 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161023 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161040 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a00e8d-f14d-4827-8a45-865982c1d36a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161197 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161259 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161237 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161263 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.161331 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.161305 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdggv\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-kube-api-access-wdggv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.197639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.197610 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7"] Oct 30 22:08:13.263953 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.263913 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.263977 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.264011 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.264036 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdggv\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-kube-api-access-wdggv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.264080 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.264129 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.264177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.264157 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a00e8d-f14d-4827-8a45-865982c1d36a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.265396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.265368 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.267689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.267657 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.268185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.268155 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.268284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.268183 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/02a00e8d-f14d-4827-8a45-865982c1d36a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.269087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.269058 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.323675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.323641 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.323844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.323820 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdggv\" (UniqueName: \"kubernetes.io/projected/02a00e8d-f14d-4827-8a45-865982c1d36a-kube-api-access-wdggv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qsnv7\" (UID: \"02a00e8d-f14d-4827-8a45-865982c1d36a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.413889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.411940 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:13.831441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.831403 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7"] Oct 30 22:08:13.838621 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:08:13.838586 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a00e8d_f14d_4827_8a45_865982c1d36a.slice/crio-b31e63a86c2eb8d159cddf8ad354995748bbabb02cdb57f8ffd947bf55e12724 WatchSource:0}: Error finding container b31e63a86c2eb8d159cddf8ad354995748bbabb02cdb57f8ffd947bf55e12724: Status 404 returned error can't find the container with id b31e63a86c2eb8d159cddf8ad354995748bbabb02cdb57f8ffd947bf55e12724 Oct 30 22:08:13.840667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.840628 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:08:13.840755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:13.840694 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:08:14.106541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.106463 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" event={"ID":"02a00e8d-f14d-4827-8a45-865982c1d36a","Type":"ContainerStarted","Data":"a4f0d7ede5b14eb6f6927d57b7af9522082efbf742e6758774cffe58e97de389"} Oct 30 22:08:14.106541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.106511 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" event={"ID":"02a00e8d-f14d-4827-8a45-865982c1d36a","Type":"ContainerStarted","Data":"b31e63a86c2eb8d159cddf8ad354995748bbabb02cdb57f8ffd947bf55e12724"} Oct 30 22:08:14.107025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.106610 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:14.108063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.108040 4305 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qsnv7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:14.108170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.108085 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" podUID="02a00e8d-f14d-4827-8a45-865982c1d36a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:14.221378 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:14.221326 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" podStartSLOduration=1.221306959 podStartE2EDuration="1.221306959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:14.219985977 +0000 UTC m=+414.815074285" watchObservedRunningTime="2026-10-30 22:08:14.221306959 +0000 UTC m=+414.816395257" Oct 30 22:08:15.111185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:15.111154 4305 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qsnv7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:08:15.111534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:15.111201 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" podUID="02a00e8d-f14d-4827-8a45-865982c1d36a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:08:16.111175 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:08:16.111141 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:08:18.111456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.111422 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qsnv7" Oct 30 22:08:18.320666 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.320633 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:08:18.320900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.320854 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" containerID="cri-o://3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f" gracePeriod=30 Oct 30 22:08:18.564498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.564475 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:08:18.608727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608699 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.608871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608739 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.608871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608815 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.608871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608851 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.609043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608897 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.609043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608916 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.609043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.608943 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\") pod \"e61f2bc4-a638-40c7-bbfe-49049b758602\" (UID: \"e61f2bc4-a638-40c7-bbfe-49049b758602\") " Oct 30 22:08:18.609205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.609161 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:18.611283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611246 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.611283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611268 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.611603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611325 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:18.611603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611407 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.611603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611441 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:18.611603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.611488 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2" pod "e61f2bc4-a638-40c7-bbfe-49049b758602" (UID: "e61f2bc4-a638-40c7-bbfe-49049b758602"). InnerVolumeSpecName "kube-api-access-s7fx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:18.709727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709701 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fx2\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-kube-api-access-s7fx2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709725 4305 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-kubeconfig\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709735 4305 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-cacerts\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709744 4305 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-dns-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709753 4305 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-csr-ca-configmap\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709762 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e61f2bc4-a638-40c7-bbfe-49049b758602-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:18.709899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:18.709770 4305 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e61f2bc4-a638-40c7-bbfe-49049b758602-local-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:19.126933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.126850 4305 generic.go:356] "Generic (PLEG): container finished" podID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerID="3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f" exitCode=0 Oct 30 22:08:19.126933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.126887 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" event={"ID":"e61f2bc4-a638-40c7-bbfe-49049b758602","Type":"ContainerDied","Data":"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f"} Oct 30 22:08:19.126933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.126893 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" Oct 30 22:08:19.126933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.126927 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf" event={"ID":"e61f2bc4-a638-40c7-bbfe-49049b758602","Type":"ContainerDied","Data":"903d552ed313633b19dcc8b2addadbadbcb925edaa08d3b285b291f8c82d41ae"} Oct 30 22:08:19.127443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.126949 4305 scope.go:122] "RemoveContainer" containerID="3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f" Oct 30 22:08:19.135443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.135426 4305 scope.go:122] "RemoveContainer" containerID="3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f" Oct 30 22:08:19.135743 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:08:19.135720 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f\": container with ID starting with 3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f not found: ID does not exist" containerID="3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f" Oct 30 22:08:19.135806 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.135753 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f"} err="failed to get container status \"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f\": rpc error: code = NotFound desc = could not find container \"3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f\": container with ID starting with 3596f6e4c9d6468a8bffa6c09d244fad8805716a27317b4b79019805ac00cb4f not found: ID does not exist" Oct 30 22:08:19.279618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.279580 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:08:19.325478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.325448 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f24xf"] Oct 30 22:08:19.519345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:19.519312 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" path="/var/lib/kubelet/pods/e61f2bc4-a638-40c7-bbfe-49049b758602/volumes" Oct 30 22:08:24.671601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.671567 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 22:08:24.672070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.671899 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" Oct 30 22:08:24.672070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.671911 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" Oct 30 22:08:24.672070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.671973 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61f2bc4-a638-40c7-bbfe-49049b758602" containerName="discovery" Oct 30 22:08:24.675555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.675534 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.760159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.760125 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.760159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.760162 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntlr\" (UniqueName: \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.760359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.760190 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.760359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.760316 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.770766 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.770739 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v8dzl\"" Oct 30 22:08:24.770766 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.770767 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:08:24.770960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.770739 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:08:24.772174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.772153 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 22:08:24.861227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.861198 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.861227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.861233 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntlr\" (UniqueName: \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.861499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.861260 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.861499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.861387 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.862144 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.862112 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.863673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.863647 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.863778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.863765 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.919056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.919020 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntlr\" (UniqueName: \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\") pod \"maas-api-d6bb6f9f5-cww72\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:24.987416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:24.987382 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:25.357951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:25.357863 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 22:08:25.364228 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:08:25.364201 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a52cc8e_17e1_4725_b642_ccae28dd7693.slice/crio-ae62c965dab0664dc68f1941ee835e8f83bc005dd46117c49da74c9ab5815c88 WatchSource:0}: Error finding container ae62c965dab0664dc68f1941ee835e8f83bc005dd46117c49da74c9ab5815c88: Status 404 returned error can't find the container with id ae62c965dab0664dc68f1941ee835e8f83bc005dd46117c49da74c9ab5815c88 Oct 30 22:08:26.155143 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:26.155107 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" event={"ID":"6a52cc8e-17e1-4725-b642-ccae28dd7693","Type":"ContainerStarted","Data":"ae62c965dab0664dc68f1941ee835e8f83bc005dd46117c49da74c9ab5815c88"} Oct 30 22:08:28.166111 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:28.166073 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" event={"ID":"6a52cc8e-17e1-4725-b642-ccae28dd7693","Type":"ContainerStarted","Data":"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6"} Oct 30 22:08:28.166540 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:28.166231 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:28.277147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:28.277094 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" podStartSLOduration=2.189102376 podStartE2EDuration="4.277077287s" totalImagesPullingTime="2.087974911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:28.276284246 +0000 UTC m=+428.871372576" watchObservedRunningTime="2026-10-30 22:08:28.277077287 +0000 UTC m=+428.872165592" Oct 30 22:08:34.176744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:08:34.176716 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 22:08:46.107997 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:08:46.107964 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:09:16.108380 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:09:16.108335 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:09:46.108782 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:09:46.108702 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:10:08.572173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:08.572138 4305 generic.go:356] "Generic (PLEG): container finished" podID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerID="60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786" exitCode=1 Oct 30 22:10:08.572563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:08.572189 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerDied","Data":"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786"} Oct 30 22:10:08.572563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:08.572503 4305 scope.go:122] "RemoveContainer" containerID="60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786" Oct 30 22:10:10.584337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:10.584303 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerStarted","Data":"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9"} Oct 30 22:10:10.584737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:10.584512 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:10:16.108690 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:10:16.108653 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:10:21.590906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:21.590866 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:10:22.648502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.648469 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c"] Oct 30 22:10:22.651944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.651922 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.747055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.747018 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 22:10:22.747055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.747044 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 22:10:22.747239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.747031 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dwlqg\"" Oct 30 22:10:22.747239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.747025 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 22:10:22.748533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.748516 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c"] Oct 30 22:10:22.774097 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774071 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7qc\" (UniqueName: \"kubernetes.io/projected/06f21021-46c6-4539-b4e0-5ce82fb489b0-kube-api-access-vh7qc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.774224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774103 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.774224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774129 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.774224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774149 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.774337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774245 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.774337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.774300 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f21021-46c6-4539-b4e0-5ce82fb489b0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875061 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875025 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875068 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875121 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875187 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f21021-46c6-4539-b4e0-5ce82fb489b0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875224 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7qc\" (UniqueName: \"kubernetes.io/projected/06f21021-46c6-4539-b4e0-5ce82fb489b0-kube-api-access-vh7qc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875549 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875567 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.875695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.875638 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.877138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.877115 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f21021-46c6-4539-b4e0-5ce82fb489b0-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.877668 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.877647 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f21021-46c6-4539-b4e0-5ce82fb489b0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.932957 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.932901 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7qc\" (UniqueName: \"kubernetes.io/projected/06f21021-46c6-4539-b4e0-5ce82fb489b0-kube-api-access-vh7qc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rqm2c\" (UID: \"06f21021-46c6-4539-b4e0-5ce82fb489b0\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:22.964109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:22.964085 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:23.332532 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:23.332499 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c"] Oct 30 22:10:23.339442 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:23.339416 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f21021_46c6_4539_b4e0_5ce82fb489b0.slice/crio-f6c22ab3d9f0c3d3feb931a3d5955d5fc74d058fe7ac35d5e31b60e22e64f451 WatchSource:0}: Error finding container f6c22ab3d9f0c3d3feb931a3d5955d5fc74d058fe7ac35d5e31b60e22e64f451: Status 404 returned error can't find the container with id f6c22ab3d9f0c3d3feb931a3d5955d5fc74d058fe7ac35d5e31b60e22e64f451 Oct 30 22:10:23.638585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:23.638499 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" event={"ID":"06f21021-46c6-4539-b4e0-5ce82fb489b0","Type":"ContainerStarted","Data":"f6c22ab3d9f0c3d3feb931a3d5955d5fc74d058fe7ac35d5e31b60e22e64f451"} Oct 30 22:10:25.653203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:25.653170 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" event={"ID":"06f21021-46c6-4539-b4e0-5ce82fb489b0","Type":"ContainerStarted","Data":"18a1fcd0a6f1d36ec8fdf3f098c641af86d552f267853ed458dc244cb4eb0453"} Oct 30 22:10:25.764788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:25.764708 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" podStartSLOduration=1.624685763 podStartE2EDuration="3.764695727s" totalImagesPullingTime="2.140009964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:25.763563301 +0000 UTC m=+546.358651607" watchObservedRunningTime="2026-10-30 22:10:25.764695727 +0000 UTC m=+546.359784033" Oct 30 22:10:25.964310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:25.964278 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:27.134739 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.134702 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv"] Oct 30 22:10:27.138120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.138100 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208561 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208761 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208606 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208761 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208670 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208761 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208718 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208761 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208748 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.208944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.208874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnbh\" (UniqueName: \"kubernetes.io/projected/1968f99b-e5ac-4169-861b-8a7d8c0787a8-kube-api-access-fwnbh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.230751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.230726 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 22:10:27.234321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.234303 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv"] Oct 30 22:10:27.309277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309249 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309283 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309307 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnbh\" (UniqueName: \"kubernetes.io/projected/1968f99b-e5ac-4169-861b-8a7d8c0787a8-kube-api-access-fwnbh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309383 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309418 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309443 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309777 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.309913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.309858 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.311542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.311522 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1968f99b-e5ac-4169-861b-8a7d8c0787a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.311820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.311785 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1968f99b-e5ac-4169-861b-8a7d8c0787a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.366895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.366865 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnbh\" (UniqueName: \"kubernetes.io/projected/1968f99b-e5ac-4169-861b-8a7d8c0787a8-kube-api-access-fwnbh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv\" (UID: \"1968f99b-e5ac-4169-861b-8a7d8c0787a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.450890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.450864 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:27.819132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:27.819058 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv"] Oct 30 22:10:27.825791 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:27.825765 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1968f99b_e5ac_4169_861b_8a7d8c0787a8.slice/crio-9f41b281669f77ea16008999ce14a731ad8b1b154c3670b044f2448a7b2b9169 WatchSource:0}: Error finding container 9f41b281669f77ea16008999ce14a731ad8b1b154c3670b044f2448a7b2b9169: Status 404 returned error can't find the container with id 9f41b281669f77ea16008999ce14a731ad8b1b154c3670b044f2448a7b2b9169 Oct 30 22:10:28.666791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:28.666749 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" event={"ID":"1968f99b-e5ac-4169-861b-8a7d8c0787a8","Type":"ContainerStarted","Data":"3540346e09209c5cdc3eddfca195e798f89dfa647bcf46cbdfb0db07fd749c94"} Oct 30 22:10:28.667183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:28.666826 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" event={"ID":"1968f99b-e5ac-4169-861b-8a7d8c0787a8","Type":"ContainerStarted","Data":"9f41b281669f77ea16008999ce14a731ad8b1b154c3670b044f2448a7b2b9169"} Oct 30 22:10:28.777976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:28.777921 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" podStartSLOduration=1.179693562 podStartE2EDuration="1.77790607s" totalImagesPullingTime="598.212508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:28.777353009 +0000 UTC m=+549.372441340" watchObservedRunningTime="2026-10-30 22:10:28.77790607 +0000 UTC m=+549.372994378" Oct 30 22:10:29.451317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:29.451273 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:31.095414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.095385 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v5lxk"] Oct 30 22:10:31.098959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.098938 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135462 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135435 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-home\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135591 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135477 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135616 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135679 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135780 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135764 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd65b990-64a6-433a-870e-f3e60fb09f37-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.135875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.135858 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbj2\" (UniqueName: \"kubernetes.io/projected/cd65b990-64a6-433a-870e-f3e60fb09f37-kube-api-access-cbbj2\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.193608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.193579 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 22:10:31.195172 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.195151 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v5lxk"] Oct 30 22:10:31.237015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.236984 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd65b990-64a6-433a-870e-f3e60fb09f37-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237034 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbj2\" (UniqueName: \"kubernetes.io/projected/cd65b990-64a6-433a-870e-f3e60fb09f37-kube-api-access-cbbj2\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237102 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-home\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237146 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237187 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237231 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237720 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-home\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237887 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.237981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.237959 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.239570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.239547 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd65b990-64a6-433a-870e-f3e60fb09f37-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.239742 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.239723 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd65b990-64a6-433a-870e-f3e60fb09f37-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.294515 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.294490 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbj2\" (UniqueName: \"kubernetes.io/projected/cd65b990-64a6-433a-870e-f3e60fb09f37-kube-api-access-cbbj2\") pod \"echo-simulator-kserve-64b6f8f96c-v5lxk\" (UID: \"cd65b990-64a6-433a-870e-f3e60fb09f37\") " pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.410929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.410853 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:31.779714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:31.779680 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-v5lxk"] Oct 30 22:10:31.785404 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:31.785375 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd65b990_64a6_433a_870e_f3e60fb09f37.slice/crio-ee64f339a55ecfe9d3eefe1faf6a9cb83150e691afc102535e85ca7acba2e44f WatchSource:0}: Error finding container ee64f339a55ecfe9d3eefe1faf6a9cb83150e691afc102535e85ca7acba2e44f: Status 404 returned error can't find the container with id ee64f339a55ecfe9d3eefe1faf6a9cb83150e691afc102535e85ca7acba2e44f Oct 30 22:10:32.684954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:32.684914 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" event={"ID":"cd65b990-64a6-433a-870e-f3e60fb09f37","Type":"ContainerStarted","Data":"ee64f339a55ecfe9d3eefe1faf6a9cb83150e691afc102535e85ca7acba2e44f"} Oct 30 22:10:32.964817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:32.964778 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:32.977765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:32.977736 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:33.701331 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:33.701300 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rqm2c" Oct 30 22:10:34.695221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:34.695188 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" event={"ID":"cd65b990-64a6-433a-870e-f3e60fb09f37","Type":"ContainerStarted","Data":"4607645ad084f9f115ce1280813a3d5e3a9cebec1f95f52342f4dc1594c942f9"} Oct 30 22:10:34.807257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:34.807197 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" podStartSLOduration=1.369881645 podStartE2EDuration="3.807180142s" totalImagesPullingTime="2.437298497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:34.805581833 +0000 UTC m=+555.400670151" watchObservedRunningTime="2026-10-30 22:10:34.807180142 +0000 UTC m=+555.402268450" Oct 30 22:10:35.411439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:35.411400 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:37.451408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:37.451376 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:37.463460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:37.463437 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:37.718791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:37.718709 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv" Oct 30 22:10:41.411844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:41.411807 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:41.424346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:41.424318 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:41.736249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:41.736218 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-v5lxk" Oct 30 22:10:45.623483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.623451 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr"] Oct 30 22:10:45.626653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.626634 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653227 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653265 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653375 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653408 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqk8\" (UniqueName: \"kubernetes.io/projected/46b614aa-46bd-4c6e-8894-8758705961d8-kube-api-access-ncqk8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653522 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.653597 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.653565 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b614aa-46bd-4c6e-8894-8758705961d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.726631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.726596 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 22:10:45.733330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.733296 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr"] Oct 30 22:10:45.755036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755004 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755057 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqk8\" (UniqueName: \"kubernetes.io/projected/46b614aa-46bd-4c6e-8894-8758705961d8-kube-api-access-ncqk8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755104 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755137 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b614aa-46bd-4c6e-8894-8758705961d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755206 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755229 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755724 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755738 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.755817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.755775 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.757354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.757332 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b614aa-46bd-4c6e-8894-8758705961d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.757748 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.757731 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b614aa-46bd-4c6e-8894-8758705961d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.815438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.815415 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqk8\" (UniqueName: \"kubernetes.io/projected/46b614aa-46bd-4c6e-8894-8758705961d8-kube-api-access-ncqk8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr\" (UID: \"46b614aa-46bd-4c6e-8894-8758705961d8\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:45.938524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:45.938461 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:46.108324 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:10:46.108290 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:10:46.311388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:46.311356 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr"] Oct 30 22:10:46.318117 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:46.318090 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b614aa_46bd_4c6e_8894_8758705961d8.slice/crio-2468da60eba1855cc1b402a49c6ead66cc92bc08b95c306799e987a31457f849 WatchSource:0}: Error finding container 2468da60eba1855cc1b402a49c6ead66cc92bc08b95c306799e987a31457f849: Status 404 returned error can't find the container with id 2468da60eba1855cc1b402a49c6ead66cc92bc08b95c306799e987a31457f849 Oct 30 22:10:46.747032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:46.746996 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" event={"ID":"46b614aa-46bd-4c6e-8894-8758705961d8","Type":"ContainerStarted","Data":"2468da60eba1855cc1b402a49c6ead66cc92bc08b95c306799e987a31457f849"} Oct 30 22:10:47.752724 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:47.752688 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" event={"ID":"46b614aa-46bd-4c6e-8894-8758705961d8","Type":"ContainerStarted","Data":"f5f31d4c0c8e378632d02679dbdbb9616622c94443e00b8a22ea6f7838effc81"} Oct 30 22:10:47.864101 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:47.864054 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" podStartSLOduration=2.278790299 podStartE2EDuration="2.864041455s" totalImagesPullingTime="585.251156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:47.862893017 +0000 UTC m=+568.457981324" watchObservedRunningTime="2026-10-30 22:10:47.864041455 +0000 UTC m=+568.459129760" Oct 30 22:10:47.939362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:47.939333 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:49.130575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.130543 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2"] Oct 30 22:10:49.134044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.134024 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182108 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182174 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182242 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182333 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182286 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182348 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djn8p\" (UniqueName: \"kubernetes.io/projected/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-kube-api-access-djn8p\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.182375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.182372 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.192262 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.192239 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 22:10:49.230146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.230119 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2"] Oct 30 22:10:49.282751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.282718 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.282938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.282765 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.282938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.282888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.282949 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.282979 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djn8p\" (UniqueName: \"kubernetes.io/projected/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-kube-api-access-djn8p\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.283001 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.283159 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.283174 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.283371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.283269 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.285182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.285159 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.285475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.285454 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.344195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.344173 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djn8p\" (UniqueName: \"kubernetes.io/projected/bed155d2-07c5-43c7-a6a9-0bc34a01e4ef-kube-api-access-djn8p\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vgnc2\" (UID: \"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.446027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.446003 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:49.817183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:49.817112 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2"] Oct 30 22:10:49.823715 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:49.823691 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed155d2_07c5_43c7_a6a9_0bc34a01e4ef.slice/crio-bbb7adba74fdcc89e11e1689f38df97881e503ab93f6d8e308bca5da0895e212 WatchSource:0}: Error finding container bbb7adba74fdcc89e11e1689f38df97881e503ab93f6d8e308bca5da0895e212: Status 404 returned error can't find the container with id bbb7adba74fdcc89e11e1689f38df97881e503ab93f6d8e308bca5da0895e212 Oct 30 22:10:50.766024 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:50.765989 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" event={"ID":"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef","Type":"ContainerStarted","Data":"7572b381b7e5c45481c6ec082b5b33bd89b230bbb1a92c9329f9d5768d3ec7a0"} Oct 30 22:10:50.766382 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:50.766052 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" event={"ID":"bed155d2-07c5-43c7-a6a9-0bc34a01e4ef","Type":"ContainerStarted","Data":"bbb7adba74fdcc89e11e1689f38df97881e503ab93f6d8e308bca5da0895e212"} Oct 30 22:10:50.879314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:50.879268 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" podStartSLOduration=1.292815377 podStartE2EDuration="1.879253748s" totalImagesPullingTime="586.438371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.878622307 +0000 UTC m=+571.473710618" watchObservedRunningTime="2026-10-30 22:10:50.879253748 +0000 UTC m=+571.474342054" Oct 30 22:10:51.446693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:51.446662 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:54.627545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.627511 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9"] Oct 30 22:10:54.631435 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.631414 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.704926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.704892 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 22:10:54.706576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.706547 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9"] Oct 30 22:10:54.728116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728090 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.728222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728141 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.728222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728208 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.728353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728338 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.728410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728392 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdspt\" (UniqueName: \"kubernetes.io/projected/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-kube-api-access-sdspt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.728478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.728459 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.828899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.828868 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.828929 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.828957 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829053 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829123 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829176 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdspt\" (UniqueName: \"kubernetes.io/projected/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-kube-api-access-sdspt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829332 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829258 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829626 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.829822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.829775 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.837032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.832124 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.837032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.832489 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.887389 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.887323 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdspt\" (UniqueName: \"kubernetes.io/projected/7f8b3232-9131-40d2-81ba-b2c3f7ec6562-kube-api-access-sdspt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5blq9\" (UID: \"7f8b3232-9131-40d2-81ba-b2c3f7ec6562\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:54.943809 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:54.943766 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:55.310260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:55.310225 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9"] Oct 30 22:10:55.316960 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:10:55.316935 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8b3232_9131_40d2_81ba_b2c3f7ec6562.slice/crio-423061f6303f77e4b2eff6689ac183f2c17c55a00e245da6b9289788652cd3c4 WatchSource:0}: Error finding container 423061f6303f77e4b2eff6689ac183f2c17c55a00e245da6b9289788652cd3c4: Status 404 returned error can't find the container with id 423061f6303f77e4b2eff6689ac183f2c17c55a00e245da6b9289788652cd3c4 Oct 30 22:10:55.788627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:55.788592 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" event={"ID":"7f8b3232-9131-40d2-81ba-b2c3f7ec6562","Type":"ContainerStarted","Data":"423061f6303f77e4b2eff6689ac183f2c17c55a00e245da6b9289788652cd3c4"} Oct 30 22:10:55.938894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:55.938866 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:55.951370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:55.951349 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:56.794214 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:56.794175 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" event={"ID":"7f8b3232-9131-40d2-81ba-b2c3f7ec6562","Type":"ContainerStarted","Data":"034dbac8867f998d82793c8dc13acfb67c6d05eff5122278944bd95720178d76"} Oct 30 22:10:56.805410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:56.805385 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr" Oct 30 22:10:56.905622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:56.905569 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" podStartSLOduration=2.340722992 podStartE2EDuration="2.905553423s" totalImagesPullingTime="564.830431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:56.9039491 +0000 UTC m=+577.499037406" watchObservedRunningTime="2026-10-30 22:10:56.905553423 +0000 UTC m=+577.500641729" Oct 30 22:10:56.944599 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:56.944566 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:10:59.446685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:59.446650 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:59.458686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:59.458654 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:10:59.819404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:10:59.819328 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vgnc2" Oct 30 22:11:04.944331 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:04.944252 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:11:04.956445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:04.956424 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:11:05.844550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:05.844519 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5blq9" Oct 30 22:11:08.705755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.705719 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:11:08.709152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.709130 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.803942 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.803909 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\"" Oct 30 22:11:08.811814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.811770 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:11:08.852410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852377 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852408 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852442 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852555 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk77p\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852610 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852635 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852680 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852765 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.852850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.852790 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.953990 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.953959 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.953998 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954017 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954050 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954071 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954113 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954258 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954302 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk77p\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954451 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954714 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954774 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954790 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.954987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.954965 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.956499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.956452 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:08.956549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:08.956532 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:09.011653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:09.011622 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:09.011769 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:09.011752 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk77p\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lhj8w\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:09.022567 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:09.022548 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:09.442713 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:09.442682 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:11:09.449195 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:11:09.449168 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6cec54_a4ca_4da9_8405_da222715a1e6.slice/crio-817d5a8f11d2c4cda0d22589434ea5423cf551d2d085d901e3125a2211829989 WatchSource:0}: Error finding container 817d5a8f11d2c4cda0d22589434ea5423cf551d2d085d901e3125a2211829989: Status 404 returned error can't find the container with id 817d5a8f11d2c4cda0d22589434ea5423cf551d2d085d901e3125a2211829989 Oct 30 22:11:09.854008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:09.853925 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" event={"ID":"4f6cec54-a4ca-4da9-8405-da222715a1e6","Type":"ContainerStarted","Data":"817d5a8f11d2c4cda0d22589434ea5423cf551d2d085d901e3125a2211829989"} Oct 30 22:11:13.417891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:13.417862 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:11:13.418158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:13.417929 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:11:13.418158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:13.417959 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:11:13.878753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:13.878719 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" event={"ID":"4f6cec54-a4ca-4da9-8405-da222715a1e6","Type":"ContainerStarted","Data":"f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a"} Oct 30 22:11:13.995785 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:13.995726 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podStartSLOduration=2.029869384 podStartE2EDuration="5.995710654s" totalImagesPullingTime="3.96584127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:13.993894596 +0000 UTC m=+594.588982902" watchObservedRunningTime="2026-10-30 22:11:13.995710654 +0000 UTC m=+594.590798959" Oct 30 22:11:14.023517 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:14.023489 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:14.024967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:14.024942 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:14.025044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:14.024992 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:15.022981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:15.022948 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:15.023357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:15.022995 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:16.023639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:16.023609 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:16.024043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:16.023669 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:16.110333 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:11:16.110293 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:11:17.023353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:17.023318 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:17.023506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:17.023376 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:18.023289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:18.023249 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:18.023697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:18.023313 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:19.023631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:19.023589 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:19.024055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:19.023650 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:19.570714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:19.570686 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:11:19.570927 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:19.570901 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:11:20.023791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:20.023758 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:11:20.024292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:20.023848 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 22:11:21.611428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:21.611390 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:21.612216 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:21.612195 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:21.613615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:11:21.613591 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:11:46.108928 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:11:46.108893 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:12:14.699150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.699112 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:12:14.702702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.702675 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.775959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.775906 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:12:14.797005 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.796976 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b13755-openshift-default-dockercfg-45x5j\"" Oct 30 22:12:14.836447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836424 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836461 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97ft\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836584 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836662 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836693 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836780 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836763 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836793 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836868 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.836897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.836892 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.937868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937882 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937908 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937929 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937949 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937971 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.937988 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938148 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97ft\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938327 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938380 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938471 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938557 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.938858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.938834 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.940005 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.939986 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.940232 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.940213 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.996356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.996290 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97ft\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:14.999036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:14.999015 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:15.014980 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.014958 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:15.448103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.448076 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:12:15.454957 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:15.454932 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324da313_68f4_4270_9984_91d39e6c15c6.slice/crio-5a70a04f123440e60897beab123909ae8dc587175858fd179c8559d0fc889fff WatchSource:0}: Error finding container 5a70a04f123440e60897beab123909ae8dc587175858fd179c8559d0fc889fff: Status 404 returned error can't find the container with id 5a70a04f123440e60897beab123909ae8dc587175858fd179c8559d0fc889fff Oct 30 22:12:15.457413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.457373 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:15.457513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.457463 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:15.457513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.457504 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:15.591895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.591861 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:12:15.594393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.594369 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.644011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.643894 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f80a0e-openshift-default-dockercfg-h5r64\"" Oct 30 22:12:15.657229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.657193 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:12:15.745171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745068 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745123 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745239 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745280 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745325 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gz4\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745365 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745436 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745490 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.745677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.745534 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846101 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846070 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846123 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846153 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846172 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846193 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846221 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846243 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846282 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gz4\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846310 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846564 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.846992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.846963 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.847108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.847010 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.847156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.847105 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.847355 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.847338 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.848551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.848530 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.848688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.848670 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.895280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.895250 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:12:15.897708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.897686 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:15.907054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.907034 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.907141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.907113 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gz4\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:15.946934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.946911 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be17e7-openshift-default-dockercfg-bb9z8\"" Oct 30 22:12:15.960279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:15.960254 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:12:16.048460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048420 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048654 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048495 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048654 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048611 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048681 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048728 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048825 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8mp\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.048931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048891 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.049010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.048939 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.110611 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:16.110575 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:12:16.149501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149468 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149505 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149531 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149557 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149607 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149636 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.149684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149675 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.149715 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8mp\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150019 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150117 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150171 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150326 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.150469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.150449 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.151688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.151657 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerStarted","Data":"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef"} Oct 30 22:12:16.151820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.151722 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerStarted","Data":"5a70a04f123440e60897beab123909ae8dc587175858fd179c8559d0fc889fff"} Oct 30 22:12:16.152350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.152323 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.152441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.152427 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.207255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.207231 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:16.211941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.211916 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.212175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.212156 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8mp\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.273161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.273099 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podStartSLOduration=2.273082042 podStartE2EDuration="2.273082042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:16.273069699 +0000 UTC m=+656.868158005" watchObservedRunningTime="2026-10-30 22:12:16.273082042 +0000 UTC m=+656.868170347" Oct 30 22:12:16.509985 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.509951 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:16.584015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.583982 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:12:16.590173 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:16.590146 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105dc63b_a219_4f9f_840f_a97f1247be5b.slice/crio-01c7f0759403ec22d4708af132d8e3bba6982a10d2684b3f0f21889987f2a2c1 WatchSource:0}: Error finding container 01c7f0759403ec22d4708af132d8e3bba6982a10d2684b3f0f21889987f2a2c1: Status 404 returned error can't find the container with id 01c7f0759403ec22d4708af132d8e3bba6982a10d2684b3f0f21889987f2a2c1 Oct 30 22:12:16.592462 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.592434 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:16.592521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.592495 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:16.592559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.592531 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:16.885852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.885761 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:12:16.892063 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:16.892035 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48eddfb_85f8_4fc4_aa5a_44dff6778940.slice/crio-36d542fc58fdb7b830e16d9a40da6bc693751b7df20990bb165d221a4a198df4 WatchSource:0}: Error finding container 36d542fc58fdb7b830e16d9a40da6bc693751b7df20990bb165d221a4a198df4: Status 404 returned error can't find the container with id 36d542fc58fdb7b830e16d9a40da6bc693751b7df20990bb165d221a4a198df4 Oct 30 22:12:16.894406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.894373 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:16.894524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.894455 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:16.894524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:16.894498 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:12:17.015474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.015440 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:17.021703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.021678 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:17.158717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.158624 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerStarted","Data":"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14"} Oct 30 22:12:17.158717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.158688 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerStarted","Data":"01c7f0759403ec22d4708af132d8e3bba6982a10d2684b3f0f21889987f2a2c1"} Oct 30 22:12:17.160381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.160351 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerStarted","Data":"cb03f68f7c008653f6ea76c5d51170582ec3312c69a54a0a1f80650de572c1d4"} Oct 30 22:12:17.160497 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.160407 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerStarted","Data":"36d542fc58fdb7b830e16d9a40da6bc693751b7df20990bb165d221a4a198df4"} Oct 30 22:12:17.160497 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.160459 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:17.162064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.162046 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:12:17.208412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.208377 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:17.213709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.213681 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:17.280437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.280362 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podStartSLOduration=2.280345518 podStartE2EDuration="2.280345518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.276991557 +0000 UTC m=+657.872079864" watchObservedRunningTime="2026-10-30 22:12:17.280345518 +0000 UTC m=+657.875433825" Oct 30 22:12:17.397460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.397398 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podStartSLOduration=2.397377793 podStartE2EDuration="2.397377793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.394763525 +0000 UTC m=+657.989851855" watchObservedRunningTime="2026-10-30 22:12:17.397377793 +0000 UTC m=+657.992466099" Oct 30 22:12:17.526938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.526910 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:17.529484 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:17.529455 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:18.164711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:18.164677 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:18.165113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:18.164735 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:18.165251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:18.165234 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:12:18.165413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:18.165394 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:12:25.745740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.745710 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:12:25.748140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.748116 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.798697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.798675 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-af30a9-serving-cert\"" Oct 30 22:12:25.843541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.843507 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.843686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.843553 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxgr\" (UniqueName: \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.843686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.843630 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.843819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.843785 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.847218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.847197 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:12:25.945025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.944990 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.945228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.945040 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.945228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.945061 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxgr\" (UniqueName: \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.945228 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:25.945202 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-serving-cert: secret "maas-api-e2e-worker-w1-af30a9-serving-cert" not found Oct 30 22:12:25.945390 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.945232 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.945390 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:25.945280 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls podName:838c9a5b-657f-4270-86ad-1b643dc837a4 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:26.445259322 +0000 UTC m=+667.040347607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls") pod "maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" (UID: "838c9a5b-657f-4270-86ad-1b643dc837a4") : secret "maas-api-e2e-worker-w1-af30a9-serving-cert" not found Oct 30 22:12:25.945990 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.945970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:25.947470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:25.947449 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:26.008710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:26.008600 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxgr\" (UniqueName: \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:26.449984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:26.449940 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:26.452310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:26.452282 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:26.660420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:26.660388 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:27.036286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:27.036206 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:12:27.042460 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:27.042432 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838c9a5b_657f_4270_86ad_1b643dc837a4.slice/crio-fb26de5e924be5c13669d9a479730550d9b05cdc1fb16edcc787aef3b64454c6 WatchSource:0}: Error finding container fb26de5e924be5c13669d9a479730550d9b05cdc1fb16edcc787aef3b64454c6: Status 404 returned error can't find the container with id fb26de5e924be5c13669d9a479730550d9b05cdc1fb16edcc787aef3b64454c6 Oct 30 22:12:27.043893 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:27.043873 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:12:27.206326 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:27.206297 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" event={"ID":"838c9a5b-657f-4270-86ad-1b643dc837a4","Type":"ContainerStarted","Data":"fb26de5e924be5c13669d9a479730550d9b05cdc1fb16edcc787aef3b64454c6"} Oct 30 22:12:28.212843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:28.212790 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" event={"ID":"838c9a5b-657f-4270-86ad-1b643dc837a4","Type":"ContainerStarted","Data":"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc"} Oct 30 22:12:28.213244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:28.212932 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:28.325057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:28.324956 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" podStartSLOduration=2.375465529 podStartE2EDuration="3.324942759s" totalImagesPullingTime="949.47723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:28.3233715 +0000 UTC m=+668.918459797" watchObservedRunningTime="2026-10-30 22:12:28.324942759 +0000 UTC m=+668.920031064" Oct 30 22:12:31.090014 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.089982 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:12:31.092458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.092434 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.099828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.099787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jf77\" (UniqueName: \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.099951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.099898 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.100000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.099960 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.100040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.100021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.144095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.144073 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-23f3d4-serving-cert\"" Oct 30 22:12:31.194315 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.194276 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:12:31.201888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.201858 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.202054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.201916 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.202054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.201976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.202186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.202078 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jf77\" (UniqueName: \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.202470 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:31.202439 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-serving-cert: secret "maas-api-e2e-worker-w3-23f3d4-serving-cert" not found Oct 30 22:12:31.202645 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:31.202629 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls podName:0e95a88c-18da-4a65-b471-20528f5a7cc6 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:31.70261016 +0000 UTC m=+672.297698448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls") pod "maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" (UID: "0e95a88c-18da-4a65-b471-20528f5a7cc6") : secret "maas-api-e2e-worker-w3-23f3d4-serving-cert" not found Oct 30 22:12:31.203253 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.203229 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.206782 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.206751 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.269202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.269172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jf77\" (UniqueName: \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.706889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.706855 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:31.709432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:31.709403 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:32.004170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:32.004063 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:32.393614 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:32.393538 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:12:32.400089 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:32.400061 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e95a88c_18da_4a65_b471_20528f5a7cc6.slice/crio-cc3a9af71f47a1cc47634234141a2ffbc89954ee9036ee63310f113d1a0c83bc WatchSource:0}: Error finding container cc3a9af71f47a1cc47634234141a2ffbc89954ee9036ee63310f113d1a0c83bc: Status 404 returned error can't find the container with id cc3a9af71f47a1cc47634234141a2ffbc89954ee9036ee63310f113d1a0c83bc Oct 30 22:12:33.237646 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:33.237611 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" event={"ID":"0e95a88c-18da-4a65-b471-20528f5a7cc6","Type":"ContainerStarted","Data":"cc3a9af71f47a1cc47634234141a2ffbc89954ee9036ee63310f113d1a0c83bc"} Oct 30 22:12:34.223373 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:34.223344 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:12:34.247371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:34.247330 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" event={"ID":"0e95a88c-18da-4a65-b471-20528f5a7cc6","Type":"ContainerStarted","Data":"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa"} Oct 30 22:12:34.247628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:34.247609 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:34.458935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:34.458884 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" podStartSLOduration=2.590946633 podStartE2EDuration="3.458868109s" totalImagesPullingTime="867.921476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.458461817 +0000 UTC m=+675.053550122" watchObservedRunningTime="2026-10-30 22:12:34.458868109 +0000 UTC m=+675.053956412" Oct 30 22:12:36.517121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.517087 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:12:36.519496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.519473 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.555300 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.555275 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.555396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.555322 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.555458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.555430 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftp4\" (UniqueName: \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.555510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.555468 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.568772 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.568750 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-eb4aa1-serving-cert\"" Oct 30 22:12:36.616867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.616832 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:12:36.656901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.656872 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.657058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.656919 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.657058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.656965 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftp4\" (UniqueName: \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.657058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.656994 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.657222 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:36.657085 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-serving-cert: secret "maas-api-e2e-worker-w2-eb4aa1-serving-cert" not found Oct 30 22:12:36.657222 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:36.657159 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls podName:a0d51899-dda4-46b8-8cba-fa6aa75279a6 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:37.157138095 +0000 UTC m=+677.752226399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls") pod "maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" (UID: "a0d51899-dda4-46b8-8cba-fa6aa75279a6") : secret "maas-api-e2e-worker-w2-eb4aa1-serving-cert" not found Oct 30 22:12:36.657684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.657661 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.659413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.659389 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:36.714227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:36.714203 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftp4\" (UniqueName: \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:37.162423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:37.162368 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:37.164703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:37.164684 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:37.431259 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:37.431184 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:37.802032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:37.801999 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:12:37.808396 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:37.808355 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d51899_dda4_46b8_8cba_fa6aa75279a6.slice/crio-dbb4ff4c316846729c0ffda3f2030924e2166b323132a2d6c4a6f798fb1f1025 WatchSource:0}: Error finding container dbb4ff4c316846729c0ffda3f2030924e2166b323132a2d6c4a6f798fb1f1025: Status 404 returned error can't find the container with id dbb4ff4c316846729c0ffda3f2030924e2166b323132a2d6c4a6f798fb1f1025 Oct 30 22:12:38.267428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:38.267395 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" event={"ID":"a0d51899-dda4-46b8-8cba-fa6aa75279a6","Type":"ContainerStarted","Data":"dbb4ff4c316846729c0ffda3f2030924e2166b323132a2d6c4a6f798fb1f1025"} Oct 30 22:12:39.275969 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:39.275932 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" event={"ID":"a0d51899-dda4-46b8-8cba-fa6aa75279a6","Type":"ContainerStarted","Data":"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f"} Oct 30 22:12:39.276462 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:39.276045 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:39.388765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:39.388717 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" podStartSLOduration=2.494189538 podStartE2EDuration="3.388703617s" totalImagesPullingTime="894.514079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:39.387120737 +0000 UTC m=+679.982209073" watchObservedRunningTime="2026-10-30 22:12:39.388703617 +0000 UTC m=+679.983791922" Oct 30 22:12:40.258295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:40.258264 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:12:45.286533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:45.286505 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:12:46.108848 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:46.108785 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:12:46.816030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.815995 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:12:46.820057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.820036 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:46.868737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.868707 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-be17e7-serving-cert\"" Oct 30 22:12:46.916911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.916880 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:12:46.955962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.955924 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:46.956168 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.955973 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:46.956168 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.956146 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79jt\" (UniqueName: \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:46.956292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:46.956209 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.057261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.057222 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79jt\" (UniqueName: \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.057460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.057280 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.057460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.057404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.057460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.057442 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.057638 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:47.057538 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-serving-cert: secret "maas-api-e2e-worker-w6-be17e7-serving-cert" not found Oct 30 22:12:47.057638 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:12:47.057619 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls podName:8ad7b205-65c7-4f23-8033-64dc828dda1b nodeName:}" failed. No retries permitted until 2026-10-30 22:12:47.557603651 +0000 UTC m=+688.152691934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls") pod "maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" (UID: "8ad7b205-65c7-4f23-8033-64dc828dda1b") : secret "maas-api-e2e-worker-w6-be17e7-serving-cert" not found Oct 30 22:12:47.058086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.058063 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.059762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.059744 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.119659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.119593 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79jt\" (UniqueName: \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.562804 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.562773 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.565130 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.565103 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") pod \"maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:47.733087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:47.733054 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:48.103297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:48.103263 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:12:48.110170 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:12:48.110146 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad7b205_65c7_4f23_8033_64dc828dda1b.slice/crio-8105679fd7930b3350e3cb464641f7879f49d000ab763b0380542a285fc0da63 WatchSource:0}: Error finding container 8105679fd7930b3350e3cb464641f7879f49d000ab763b0380542a285fc0da63: Status 404 returned error can't find the container with id 8105679fd7930b3350e3cb464641f7879f49d000ab763b0380542a285fc0da63 Oct 30 22:12:48.322225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:48.322189 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" event={"ID":"8ad7b205-65c7-4f23-8033-64dc828dda1b","Type":"ContainerStarted","Data":"8105679fd7930b3350e3cb464641f7879f49d000ab763b0380542a285fc0da63"} Oct 30 22:12:49.328102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:49.328018 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" event={"ID":"8ad7b205-65c7-4f23-8033-64dc828dda1b","Type":"ContainerStarted","Data":"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91"} Oct 30 22:12:49.328469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:49.328158 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:12:49.440854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:49.440772 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" podStartSLOduration=2.569495153 podStartE2EDuration="3.440758081s" totalImagesPullingTime="871.262928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:49.439696244 +0000 UTC m=+690.034784551" watchObservedRunningTime="2026-10-30 22:12:49.440758081 +0000 UTC m=+690.035846386" Oct 30 22:12:55.337747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:12:55.337716 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:13:04.206034 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:04.205927 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:13:04.206034 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:04.206000 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:13:04.206533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:04.206062 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:04.206533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:04.206010 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:13:06.435141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.435043 4305 generic.go:356] "Generic (PLEG): container finished" podID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerID="70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14" exitCode=0 Oct 30 22:13:06.435141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.435117 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerDied","Data":"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14"} Oct 30 22:13:06.435682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.435533 4305 scope.go:122] "RemoveContainer" containerID="70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14" Oct 30 22:13:06.436875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.436851 4305 generic.go:356] "Generic (PLEG): container finished" podID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerID="cb03f68f7c008653f6ea76c5d51170582ec3312c69a54a0a1f80650de572c1d4" exitCode=0 Oct 30 22:13:06.436963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.436918 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerDied","Data":"cb03f68f7c008653f6ea76c5d51170582ec3312c69a54a0a1f80650de572c1d4"} Oct 30 22:13:06.437020 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.436996 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:06.437096 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.437083 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:06.437134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.437118 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:06.437244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.437231 4305 scope.go:122] "RemoveContainer" containerID="cb03f68f7c008653f6ea76c5d51170582ec3312c69a54a0a1f80650de572c1d4" Oct 30 22:13:06.438494 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.438469 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:06.438564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.438543 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:06.438602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:06.438589 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:07.443399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:07.443367 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerStarted","Data":"3cd196a0aebdc6817b60c05bafd12eb6f1f2eca3e6ef54eccf0671a49e2d984d"} Oct 30 22:13:07.445287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:07.445265 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerStarted","Data":"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549"} Oct 30 22:13:07.510936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:07.510902 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:13:07.510936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:07.510917 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:07.511191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:07.510971 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:08.210823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:08.210210 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:13:08.210823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:08.210338 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 22:13:08.210823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:08.210394 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 22:13:08.515163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:08.515072 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:08.515718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:08.515685 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:09.213821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:09.213193 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 22:13:09.213821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:09.213255 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 22:13:09.523527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:09.523429 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:09.527121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:09.526428 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:10.211262 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:10.211223 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 22:13:10.212261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:10.212229 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 22:13:10.518281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:10.518190 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:10.518534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:10.518500 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:11.211824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:11.211191 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 22:13:11.211824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:11.211259 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 22:13:11.515078 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:11.514958 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:11.515339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:11.515311 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:12.213818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.213473 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 22:13:12.213818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.213532 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 22:13:12.396823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.392267 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:13:12.403482 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.403453 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.417818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.417703 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.417818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.417753 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.418040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.417837 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.418040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.417882 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpczj\" (UniqueName: \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.465870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.465318 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:13:12.529010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.528964 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d2d4c3-serving-cert\"" Oct 30 22:13:12.534951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.531502 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.535223 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:13:12.531587 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-serving-cert: secret "maas-api-e2e-worker-w0-d2d4c3-serving-cert" not found Oct 30 22:13:12.535439 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:13:12.535425 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls podName:ca71e3e9-ada3-4096-a730-504f508521cf nodeName:}" failed. No retries permitted until 2026-10-30 22:13:13.035384522 +0000 UTC m=+713.630472809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls") pod "maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" (UID: "ca71e3e9-ada3-4096-a730-504f508521cf") : secret "maas-api-e2e-worker-w0-d2d4c3-serving-cert" not found Oct 30 22:13:12.535840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.531926 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 22:13:12.536013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.535988 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 22:13:12.536184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.535185 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.537436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.536496 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.537670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.537651 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpczj\" (UniqueName: \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.537898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.537682 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.542628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.542603 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:12.645519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:12.645325 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpczj\" (UniqueName: \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:13.064821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.054349 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:13.072307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.072272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:13.211755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.211719 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:13:13.212142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.212123 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:13:13.212753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.212738 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:13:13.333319 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.333247 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:13.518202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.518170 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:13:13.518618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.518590 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:13:13.519450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.519422 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:13:13.772718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:13.772568 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:13:13.783812 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:13:13.783765 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca71e3e9_ada3_4096_a730_504f508521cf.slice/crio-b7d4467031dfef75b6a500e2c38e36e16a95e3e1d2a1093b1653de975ab6f5a2 WatchSource:0}: Error finding container b7d4467031dfef75b6a500e2c38e36e16a95e3e1d2a1093b1653de975ab6f5a2: Status 404 returned error can't find the container with id b7d4467031dfef75b6a500e2c38e36e16a95e3e1d2a1093b1653de975ab6f5a2 Oct 30 22:13:14.553537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:14.553494 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" event={"ID":"ca71e3e9-ada3-4096-a730-504f508521cf","Type":"ContainerStarted","Data":"b7d4467031dfef75b6a500e2c38e36e16a95e3e1d2a1093b1653de975ab6f5a2"} Oct 30 22:13:15.564318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:15.564107 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" event={"ID":"ca71e3e9-ada3-4096-a730-504f508521cf","Type":"ContainerStarted","Data":"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076"} Oct 30 22:13:15.571220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:15.564977 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:15.684915 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:15.681508 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" podStartSLOduration=2.892002281 podStartE2EDuration="3.681485458s" totalImagesPullingTime="789.483177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:15.676752885 +0000 UTC m=+716.271841193" watchObservedRunningTime="2026-10-30 22:13:15.681485458 +0000 UTC m=+716.276573766" Oct 30 22:13:16.109440 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:13:16.109394 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:13:16.570124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.570043 4305 generic.go:356] "Generic (PLEG): container finished" podID="324da313-68f4-4270-9984-91d39e6c15c6" containerID="adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef" exitCode=0 Oct 30 22:13:16.570524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.570119 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerDied","Data":"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef"} Oct 30 22:13:16.570633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.570618 4305 scope.go:122] "RemoveContainer" containerID="adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef" Oct 30 22:13:16.571758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.571732 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:16.571862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.571818 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:16.571862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:16.571852 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:13:17.163822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:17.160895 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:13:17.577137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:17.577103 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerStarted","Data":"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79"} Oct 30 22:13:18.015239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:18.015205 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:13:18.016610 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:18.016578 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:18.016747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:18.016651 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:13:19.016234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:19.016197 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:19.016629 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:19.016265 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:13:20.015533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:20.015496 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:20.015711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:20.015549 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:13:21.015493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:21.015452 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:21.016025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:21.015518 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:13:21.578437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:21.578409 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:13:22.015586 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:22.015549 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 22:13:22.015999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:22.015612 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 22:13:23.026549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:23.023609 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:13:23.026549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:23.024522 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:13:23.027388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:13:23.027364 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:13:46.109043 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:13:46.109005 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:14:12.260356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.260269 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:14:12.262885 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.262864 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317245 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317293 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317454 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317560 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317613 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.317709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.317690 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszj2\" (UniqueName: \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.358459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.358423 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"simulator-w6-be17e7-kserve-self-signed-certs\"" Oct 30 22:14:12.358459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.358434 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"kube-root-ca.crt\"" Oct 30 22:14:12.358459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.358451 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"default-dockercfg-vw7ld\"" Oct 30 22:14:12.358690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.358500 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"openshift-service-ca.crt\"" Oct 30 22:14:12.362213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.362190 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:14:12.419225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419342 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419393 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419423 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszj2\" (UniqueName: \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419485 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419704 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419680 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419704 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419696 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.419843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419734 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.420007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.419989 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.421789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.421765 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.421987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.421971 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.477110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.477074 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszj2\" (UniqueName: \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\") pod \"simulator-w6-be17e7-kserve-66dc55c8f4-6h694\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.584875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.584765 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:12.952359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:12.952327 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:14:12.958620 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:14:12.958592 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod998636d1_6870_4142_becc_d4b6c74795ce.slice/crio-c49dbb770c5b7ed2441f92076fd38dbd1229dc02d4f7efe737cbade3532595fa WatchSource:0}: Error finding container c49dbb770c5b7ed2441f92076fd38dbd1229dc02d4f7efe737cbade3532595fa: Status 404 returned error can't find the container with id c49dbb770c5b7ed2441f92076fd38dbd1229dc02d4f7efe737cbade3532595fa Oct 30 22:14:13.847658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:13.847625 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" event={"ID":"998636d1-6870-4142-becc-d4b6c74795ce","Type":"ContainerStarted","Data":"875ecaf05942dac8e922b58702d274c18d25aed163d37c49b4e9f0688314c4bf"} Oct 30 22:14:13.848062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:13.847667 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" event={"ID":"998636d1-6870-4142-becc-d4b6c74795ce","Type":"ContainerStarted","Data":"c49dbb770c5b7ed2441f92076fd38dbd1229dc02d4f7efe737cbade3532595fa"} Oct 30 22:14:13.967310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:13.967247 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podStartSLOduration=1.96723164 podStartE2EDuration="1.96723164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:13.966961072 +0000 UTC m=+774.562049377" watchObservedRunningTime="2026-10-30 22:14:13.96723164 +0000 UTC m=+774.562319947" Oct 30 22:14:14.585900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:14.585866 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:16.108508 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:14:16.108476 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:14:16.340616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.340587 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:14:16.343523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.343500 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.392947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.392877 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23f3d4-openshift-default-dockercfg-c5fxn\"" Oct 30 22:14:16.447008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.446979 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:14:16.459367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459338 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459373 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459441 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459470 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459503 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459616 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459655 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459737 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrls\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.459900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.459787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561363 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561454 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561481 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561529 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561671 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561767 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561828 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrls\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.561944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.561865 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.562048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.562019 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.562139 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.562119 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.562213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.562146 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.562357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.562329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.562491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.562430 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.564057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.564037 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.564193 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.564177 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.619116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.619081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.619272 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.619251 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrls\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:16.655941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:16.655860 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:17.088568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.088527 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:14:17.092668 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:14:17.092640 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09df02e4_d0fd_4186_b605_02932d5739d1.slice/crio-349ca584dbc3710f1977a713e7890552dd29ea65ce23ea63cd241519d616e7a3 WatchSource:0}: Error finding container 349ca584dbc3710f1977a713e7890552dd29ea65ce23ea63cd241519d616e7a3: Status 404 returned error can't find the container with id 349ca584dbc3710f1977a713e7890552dd29ea65ce23ea63cd241519d616e7a3 Oct 30 22:14:17.095095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.095059 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:17.095180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.095148 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:17.095216 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.095183 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:17.875075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.875029 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" event={"ID":"09df02e4-d0fd-4186-b605-02932d5739d1","Type":"ContainerStarted","Data":"ff2db3badaaad33ce1b102cc6473b51bdcca42488ffd5425215d47e18c615e52"} Oct 30 22:14:17.875588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.875106 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" event={"ID":"09df02e4-d0fd-4186-b605-02932d5739d1","Type":"ContainerStarted","Data":"349ca584dbc3710f1977a713e7890552dd29ea65ce23ea63cd241519d616e7a3"} Oct 30 22:14:17.991421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:17.991369 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podStartSLOduration=1.991354742 podStartE2EDuration="1.991354742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:17.989407344 +0000 UTC m=+778.584495652" watchObservedRunningTime="2026-10-30 22:14:17.991354742 +0000 UTC m=+778.586443050" Oct 30 22:14:18.661827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:18.658253 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:18.661827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:18.658397 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:14:18.661827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:18.658443 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:14:19.669820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:19.667192 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:14:19.669820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:19.667252 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:14:20.670822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:20.662143 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:14:20.670822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:20.662201 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:14:21.661824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:21.657160 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:14:21.661824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:21.657223 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:14:22.585478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:22.585445 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:22.597945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:22.597915 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:22.660940 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:22.660904 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 22:14:22.661215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:22.661184 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 22:14:22.916718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:22.916635 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:14:23.661891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:23.661863 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:23.662289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:23.662142 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:23.662790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:23.662775 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:14:45.635505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.635473 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:14:45.638198 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.638181 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.687254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.687231 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-af30a9-openshift-default-dockercfg-fzzwk\"" Oct 30 22:14:45.743172 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.743143 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:14:45.761348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761308 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761363 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761419 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761523 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761577 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761690 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761735 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761793 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn229\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.761979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.761941 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.862748 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862716 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.862960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862764 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.862960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862785 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.862960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862830 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn229\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.862960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862922 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.862976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863013 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863044 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863077 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863239 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863252 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863383 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863395 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.863712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.863694 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.865702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.865672 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.865880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.865860 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.924447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.924372 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.924598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.924539 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn229\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:45.950972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:45.950949 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:46.108388 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:14:46.108354 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:14:46.372530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:46.372495 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:14:46.378350 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:14:46.378312 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5deb917_8d4c_4fbe_b863_40314d06facd.slice/crio-04639d3156ef75f3b85635bcacaf0bec2ebb28f8ab7905451ce67926e1063e7e WatchSource:0}: Error finding container 04639d3156ef75f3b85635bcacaf0bec2ebb28f8ab7905451ce67926e1063e7e: Status 404 returned error can't find the container with id 04639d3156ef75f3b85635bcacaf0bec2ebb28f8ab7905451ce67926e1063e7e Oct 30 22:14:46.380749 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:46.380718 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:46.380860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:46.380785 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:46.380860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:46.380836 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:14:47.022364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.022326 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" event={"ID":"b5deb917-8d4c-4fbe-b863-40314d06facd","Type":"ContainerStarted","Data":"31aefe07c5a806d8ad49fc2b790ef796c9e6cc5a4642203f4bcd179759c8a59f"} Oct 30 22:14:47.022760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.022380 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" event={"ID":"b5deb917-8d4c-4fbe-b863-40314d06facd","Type":"ContainerStarted","Data":"04639d3156ef75f3b85635bcacaf0bec2ebb28f8ab7905451ce67926e1063e7e"} Oct 30 22:14:47.140423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.140362 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podStartSLOduration=2.140342728 podStartE2EDuration="2.140342728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:47.14013189 +0000 UTC m=+807.735220220" watchObservedRunningTime="2026-10-30 22:14:47.140342728 +0000 UTC m=+807.735431037" Oct 30 22:14:47.953480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.953444 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:47.953880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.953853 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 22:14:47.954062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:47.954036 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 22:14:48.964824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:48.963323 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 22:14:48.964824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:48.963408 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 22:14:49.962822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:49.955167 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 22:14:49.962822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:49.955228 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 22:14:50.966821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:50.966190 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 22:14:50.966821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:50.966268 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 22:14:51.970983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:51.970944 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 22:14:51.971483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:51.971029 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 22:14:53.130026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:53.129984 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:53.130603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:53.130568 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:14:53.132126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:14:53.132098 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:15:16.109578 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:15:16.109538 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:15:18.362744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.362708 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:15:18.365543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.365517 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.414779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.414746 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"openshift-service-ca.crt\"" Oct 30 22:15:18.440502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440466 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnm4\" (UniqueName: \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.440676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440520 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.440676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440620 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.440676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440645 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.440831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440683 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.440831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.440742 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.460657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.460630 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"premium-w2-eb4aa1-kserve-self-signed-certs\"" Oct 30 22:15:18.460657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.460632 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"kube-root-ca.crt\"" Oct 30 22:15:18.460884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.460738 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"default-dockercfg-4bjl8\"" Oct 30 22:15:18.463131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.463105 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:15:18.542122 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542084 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542184 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnm4\" (UniqueName: \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542234 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542291 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542315 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542391 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542464 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542678 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.542777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.542746 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.544434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.544414 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.544692 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.544676 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.600730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.600699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnm4\" (UniqueName: \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\") pod \"premium-w2-eb4aa1-kserve-85db4d47bb-skxpz\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:18.676912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:18.676828 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:19.049911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:19.049875 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:15:19.055844 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:15:19.055815 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf198148e_24d1_4355_93b8_c48af59929a9.slice/crio-7170209524d7a3b2e79003624ded71018f2ecaca12e0004d8d53001b5cdd3c1f WatchSource:0}: Error finding container 7170209524d7a3b2e79003624ded71018f2ecaca12e0004d8d53001b5cdd3c1f: Status 404 returned error can't find the container with id 7170209524d7a3b2e79003624ded71018f2ecaca12e0004d8d53001b5cdd3c1f Oct 30 22:15:19.210238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:19.210202 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" event={"ID":"f198148e-24d1-4355-93b8-c48af59929a9","Type":"ContainerStarted","Data":"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af"} Oct 30 22:15:19.210411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:19.210250 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" event={"ID":"f198148e-24d1-4355-93b8-c48af59929a9","Type":"ContainerStarted","Data":"7170209524d7a3b2e79003624ded71018f2ecaca12e0004d8d53001b5cdd3c1f"} Oct 30 22:15:19.319957 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:19.319832 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podStartSLOduration=1.319780599 podStartE2EDuration="1.319780599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:19.319517344 +0000 UTC m=+839.914605650" watchObservedRunningTime="2026-10-30 22:15:19.319780599 +0000 UTC m=+839.914868922" Oct 30 22:15:19.677393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:19.677307 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:28.677998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:28.677967 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:28.690383 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:28.690359 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:29.269627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:15:29.269599 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:15:46.110181 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:15:46.110092 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:16:15.697268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.697235 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:16:15.701069 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.701039 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.750279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.750244 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"kube-root-ca.crt\"" Oct 30 22:16:15.750279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.750259 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"openshift-service-ca.crt\"" Oct 30 22:16:15.796368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.796332 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"default-dockercfg-6l7bq\"" Oct 30 22:16:15.796541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.796418 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"premium-w0-d2d4c3-kserve-self-signed-certs\"" Oct 30 22:16:15.798924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.798902 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:16:15.870648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870609 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.870873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870651 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts5l\" (UniqueName: \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.870873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870681 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.870873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870847 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.871021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870953 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.871021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.870995 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972491 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972575 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972611 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972651 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972780 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts5l\" (UniqueName: \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.972967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.972839 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.973075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.973035 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.973135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.973077 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.973207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.973189 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.974854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.974834 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:15.975310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:15.975289 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:16.030428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:16.030389 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts5l\" (UniqueName: \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\") pod \"premium-w0-d2d4c3-kserve-58d4dc4466-n4884\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:16.109064 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:16:16.109026 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:16:16.314119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:16.314034 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:16.688362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:16.688281 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:16:16.694152 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:16:16.694124 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda500a836_9f38_40d6_a0da_7ffac0708190.slice/crio-ae9cb9d7a374761c58677c767a968474c1f6feb23541725fc51e682e91a2d211 WatchSource:0}: Error finding container ae9cb9d7a374761c58677c767a968474c1f6feb23541725fc51e682e91a2d211: Status 404 returned error can't find the container with id ae9cb9d7a374761c58677c767a968474c1f6feb23541725fc51e682e91a2d211 Oct 30 22:16:17.528227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:17.528192 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" event={"ID":"a500a836-9f38-40d6-a0da-7ffac0708190","Type":"ContainerStarted","Data":"8491d7d852a1571c7c47d81d6eb7b15c2b014befaee73ef1d300619c23f153b4"} Oct 30 22:16:17.528641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:17.528232 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" event={"ID":"a500a836-9f38-40d6-a0da-7ffac0708190","Type":"ContainerStarted","Data":"ae9cb9d7a374761c58677c767a968474c1f6feb23541725fc51e682e91a2d211"} Oct 30 22:16:17.643326 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:17.643274 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podStartSLOduration=2.643259583 podStartE2EDuration="2.643259583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:17.64103511 +0000 UTC m=+898.236123428" watchObservedRunningTime="2026-10-30 22:16:17.643259583 +0000 UTC m=+898.238347889" Oct 30 22:16:18.314324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:18.314292 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:19.604874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:19.604840 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:16:19.605858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:19.605836 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:16:26.314587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:26.314544 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:26.326900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:26.326871 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:26.590264 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:16:26.590175 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:16:46.109217 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:16:46.109180 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:17:02.201661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.201585 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:17:02.205914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.205892 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.255286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.255264 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eb4aa1-openshift-default-dockercfg-l8phj\"" Oct 30 22:17:02.262088 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262058 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262098 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262156 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262246 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz85\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262275 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262290 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262400 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262315 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262433 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.262486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.262471 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.308542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.308519 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:17:02.363189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363155 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz85\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363202 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363217 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363241 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363267 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363302 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363362 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363416 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363620 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363674 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363824 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.363931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.363906 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.364133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.364110 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.364174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.364132 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.365716 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.365688 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.365876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.365855 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.421180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.421150 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.421353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.421326 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz85\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.519297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.519209 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:02.951321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.951290 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:17:02.956734 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:17:02.956707 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3d1b90_8506_493c_bc2a_3e79d21bba50.slice/crio-6858e09498726a02d46dce55fce2ac3c9345f67e58b3cf5331f96bc4f9f26e41 WatchSource:0}: Error finding container 6858e09498726a02d46dce55fce2ac3c9345f67e58b3cf5331f96bc4f9f26e41: Status 404 returned error can't find the container with id 6858e09498726a02d46dce55fce2ac3c9345f67e58b3cf5331f96bc4f9f26e41 Oct 30 22:17:02.959034 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.958998 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:02.959134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.959080 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:02.959134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:02.959127 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:03.778750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:03.778713 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" event={"ID":"3f3d1b90-8506-493c-bc2a-3e79d21bba50","Type":"ContainerStarted","Data":"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c"} Oct 30 22:17:03.779283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:03.778766 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" event={"ID":"3f3d1b90-8506-493c-bc2a-3e79d21bba50","Type":"ContainerStarted","Data":"6858e09498726a02d46dce55fce2ac3c9345f67e58b3cf5331f96bc4f9f26e41"} Oct 30 22:17:03.940927 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:03.940873 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podStartSLOduration=1.940859321 podStartE2EDuration="1.940859321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:03.939094357 +0000 UTC m=+944.534182677" watchObservedRunningTime="2026-10-30 22:17:03.940859321 +0000 UTC m=+944.535947626" Oct 30 22:17:04.520439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:04.520402 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:04.521710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:04.521683 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:17:04.521848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:04.521729 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:17:05.520057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:05.520023 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:17:05.520459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:05.520073 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:17:06.520222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:06.520187 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:17:06.520623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:06.520248 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:17:07.520012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:07.519981 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:17:07.520200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:07.520039 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:17:08.520060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:08.520024 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:17:08.520444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:08.520083 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 22:17:09.550095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:09.550062 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:09.551073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:09.551039 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:09.551764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:09.551744 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:17:16.109003 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:17:16.108963 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:17:27.984740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:27.984706 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:17:27.988848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:27.988824 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070465 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070524 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070650 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070774 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070813 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5swb\" (UniqueName: \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.070938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.070851 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.083436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.083415 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"distinct-w2-eb4aa1-kserve-self-signed-certs\"" Oct 30 22:17:28.087432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.087407 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:17:28.171557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171520 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.171729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171584 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.171729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171670 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.171729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171699 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5swb\" (UniqueName: \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.171868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171742 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.171868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.171782 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.172203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.172172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.172352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.172269 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.172352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.172292 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.174011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.173993 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.174281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.174264 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.229321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.229289 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5swb\" (UniqueName: \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\") pod \"distinct-w2-eb4aa1-kserve-748c95fb49-rpltd\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.300825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.300727 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:28.673021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.672947 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:17:28.678646 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:17:28.678615 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb180a4_1816_40e6_90b8_631bddf90c2d.slice/crio-ef835c1978c62be6a2a105233a96669e884a73812c4a2f3247480e2f5c60c1c1 WatchSource:0}: Error finding container ef835c1978c62be6a2a105233a96669e884a73812c4a2f3247480e2f5c60c1c1: Status 404 returned error can't find the container with id ef835c1978c62be6a2a105233a96669e884a73812c4a2f3247480e2f5c60c1c1 Oct 30 22:17:28.912768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.912726 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" event={"ID":"4bb180a4-1816-40e6-90b8-631bddf90c2d","Type":"ContainerStarted","Data":"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566"} Oct 30 22:17:28.912934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:28.912781 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" event={"ID":"4bb180a4-1816-40e6-90b8-631bddf90c2d","Type":"ContainerStarted","Data":"ef835c1978c62be6a2a105233a96669e884a73812c4a2f3247480e2f5c60c1c1"} Oct 30 22:17:29.029789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:29.029740 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podStartSLOduration=2.029727595 podStartE2EDuration="2.029727595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:29.027892671 +0000 UTC m=+969.622980976" watchObservedRunningTime="2026-10-30 22:17:29.029727595 +0000 UTC m=+969.624816277" Oct 30 22:17:29.301137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:29.301040 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:38.301332 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:38.301294 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:38.313489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:38.313461 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:38.981412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:38.981379 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:17:41.634156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.634125 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:17:41.638227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.638208 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.729937 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.729906 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.729953 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.729992 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730112 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730185 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730210 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730305 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730249 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730331 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgvl\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.730400 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.730369 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.741824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.741782 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:17:41.831401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831367 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831426 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgvl\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831483 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831547 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831585 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831612 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831656 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831706 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831741 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.831970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.831905 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.832127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.832053 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.832127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.832106 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.832203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.832172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.832429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.832402 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.834087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.834060 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.834346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.834323 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.889537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.889466 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.889672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.889588 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgvl\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\") pod \"maas-default-gateway-openshift-default-75bf867b6c-clhnh\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:41.951998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:41.951962 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:42.378789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.378753 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:17:42.384011 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:17:42.383979 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863ba88e_0df6_4782_bcd4_1ca3813b3d0a.slice/crio-a21b5bfd980ded17b6c98f9b35d5a59dfd6034ed4d4121d7691d02b98fe2eee9 WatchSource:0}: Error finding container a21b5bfd980ded17b6c98f9b35d5a59dfd6034ed4d4121d7691d02b98fe2eee9: Status 404 returned error can't find the container with id a21b5bfd980ded17b6c98f9b35d5a59dfd6034ed4d4121d7691d02b98fe2eee9 Oct 30 22:17:42.386383 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.386348 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.386473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.386418 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.386473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.386454 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.996224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.996189 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" event={"ID":"863ba88e-0df6-4782-bcd4-1ca3813b3d0a","Type":"ContainerStarted","Data":"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b"} Oct 30 22:17:42.996612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:42.996251 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" event={"ID":"863ba88e-0df6-4782-bcd4-1ca3813b3d0a","Type":"ContainerStarted","Data":"a21b5bfd980ded17b6c98f9b35d5a59dfd6034ed4d4121d7691d02b98fe2eee9"} Oct 30 22:17:43.121146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:43.121097 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podStartSLOduration=2.121083615 podStartE2EDuration="2.121083615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:43.11987602 +0000 UTC m=+983.714964331" watchObservedRunningTime="2026-10-30 22:17:43.121083615 +0000 UTC m=+983.716171920" Oct 30 22:17:43.968457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:43.968421 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:43.968883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:43.968856 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:43.969062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:43.969035 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:44.954189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:44.954156 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:44.954808 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:44.954759 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:45.965854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:45.962540 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:45.965854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:45.962610 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:46.108824 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:17:46.108776 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:17:46.965822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:46.962449 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:46.965822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:46.962512 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:47.960830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:47.960157 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:47.960830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:47.960261 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:48.953189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:48.953145 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-clhnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:48.953603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:48.953212 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 22:17:50.739137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:50.739105 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:50.739661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:50.739638 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:17:50.740351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:17:50.740334 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:18:02.165762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.165724 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:18:02.171321 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.171300 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.234139 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.234107 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:18:02.309372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309334 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309515 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309381 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzzj\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309510 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309546 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309620 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309646 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309716 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309705 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309731 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.309790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.309757 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411249 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411311 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411335 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzzj\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411377 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411403 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411447 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411482 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411527 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.411577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411558 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.412605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411850 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.412605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411873 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.412605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.411896 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.412605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.412007 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.412864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.412843 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.413822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.413786 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.413822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.413766 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.472953 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.472920 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzzj\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.473253 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.473228 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.485115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.485093 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:02.910098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.910022 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:18:02.950858 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:18:02.950823 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7804d7_e82e_4db3_b061_48de68a4db8e.slice/crio-fa7e2a2616c37ecea3e46786a8306e9a1ee4e5c26e24bdf17b7efa8dacae8947 WatchSource:0}: Error finding container fa7e2a2616c37ecea3e46786a8306e9a1ee4e5c26e24bdf17b7efa8dacae8947: Status 404 returned error can't find the container with id fa7e2a2616c37ecea3e46786a8306e9a1ee4e5c26e24bdf17b7efa8dacae8947 Oct 30 22:18:02.953232 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.953200 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:02.953309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.953269 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:02.953365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:02.953313 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:03.118683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:03.118580 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" event={"ID":"2c7804d7-e82e-4db3-b061-48de68a4db8e","Type":"ContainerStarted","Data":"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791"} Oct 30 22:18:03.118683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:03.118647 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" event={"ID":"2c7804d7-e82e-4db3-b061-48de68a4db8e","Type":"ContainerStarted","Data":"fa7e2a2616c37ecea3e46786a8306e9a1ee4e5c26e24bdf17b7efa8dacae8947"} Oct 30 22:18:03.235984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:03.235932 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podStartSLOduration=1.2359170640000001 podStartE2EDuration="1.235917064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:03.234578518 +0000 UTC m=+1003.829666836" watchObservedRunningTime="2026-10-30 22:18:03.235917064 +0000 UTC m=+1003.831005370" Oct 30 22:18:03.486129 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:03.486057 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:04.491550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:04.491511 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:04.492067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:04.491582 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:05.486431 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:05.486393 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:05.486597 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:05.486474 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:06.486452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:06.486417 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:06.486863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:06.486468 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:07.485623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:07.485585 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:07.485841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:07.485648 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:08.486362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:08.486327 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:08.486840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:08.486396 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:09.485888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:09.485854 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:18:09.486135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:09.485911 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 22:18:10.490366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:10.490339 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:10.490824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:10.490621 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:10.491429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:10.491412 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:18:16.111161 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:18:16.111128 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:18:19.547316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.547286 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:18:19.550940 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.550922 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.640763 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.640728 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.640942 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.640883 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.640993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.640944 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djw9s\" (UniqueName: \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.641049 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.641008 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.641108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.641086 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.641146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.641113 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.645759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.645735 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d2d4c3\"/\"distinct-w0-d2d4c3-kserve-self-signed-certs\"" Oct 30 22:18:19.648720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.648701 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:18:19.742514 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.742470 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.749665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.749576 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djw9s\" (UniqueName: \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.749883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.749685 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.749883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.749750 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.749883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.749782 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.750074 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.743178 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.751628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.750150 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.751628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.750347 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.751628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.750635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.756257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.756230 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.758205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.758181 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.820820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.820736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djw9s\" (UniqueName: \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\") pod \"distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:19.864635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:19.864609 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:20.239137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:20.239107 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:18:20.244366 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:18:20.244337 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14f0f1f8_74d1_489c_953d_3ba8e4795494.slice/crio-7dbd4c9c7ce2dd5fba07b19cab7c3c4bddd89f400c1e997fbb185de5006804b3 WatchSource:0}: Error finding container 7dbd4c9c7ce2dd5fba07b19cab7c3c4bddd89f400c1e997fbb185de5006804b3: Status 404 returned error can't find the container with id 7dbd4c9c7ce2dd5fba07b19cab7c3c4bddd89f400c1e997fbb185de5006804b3 Oct 30 22:18:21.217361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:21.217314 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" event={"ID":"14f0f1f8-74d1-489c-953d-3ba8e4795494","Type":"ContainerStarted","Data":"1746ac60b03dfffea6539d493a1c22b5be724cfbc9eddeefe71e7578ef27c455"} Oct 30 22:18:21.217749 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:21.217369 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" event={"ID":"14f0f1f8-74d1-489c-953d-3ba8e4795494","Type":"ContainerStarted","Data":"7dbd4c9c7ce2dd5fba07b19cab7c3c4bddd89f400c1e997fbb185de5006804b3"} Oct 30 22:18:21.330419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:21.330370 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podStartSLOduration=2.330354967 podStartE2EDuration="2.330354967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:21.328253761 +0000 UTC m=+1021.923342069" watchObservedRunningTime="2026-10-30 22:18:21.330354967 +0000 UTC m=+1021.925443272" Oct 30 22:18:21.865042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:21.865012 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:29.865149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:29.865121 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:29.877505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:29.877482 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:30.292852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:30.292790 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:18:45.438816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.438718 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:18:45.442667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.442647 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.487997 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28ws\" (UniqueName: \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.488083 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.488132 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.488158 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.488223 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.488338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.488265 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.492044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.492022 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"openshift-service-ca.crt\"" Oct 30 22:18:45.537734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.537712 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"kube-root-ca.crt\"" Oct 30 22:18:45.537862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.537845 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"default-dockercfg-2zk59\"" Oct 30 22:18:45.537915 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.537891 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"distinct-w4-b13755-kserve-self-signed-certs\"" Oct 30 22:18:45.540635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.540614 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.598686 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.598768 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.598931 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28ws\" (UniqueName: \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.599088 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.599143 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.599170 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.600917 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.602475 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.616821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.602727 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.624828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.623726 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.640309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.640272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.658887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.658853 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28ws\" (UniqueName: \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\") pod \"distinct-w4-b13755-kserve-7d78b686b4-nc8kx\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:45.754313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:45.754244 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:46.109102 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:18:46.109015 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:18:46.148203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:46.148176 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:18:46.154266 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:18:46.154234 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b4b704b_1b8e_48f1_a515_bc7230b8fb89.slice/crio-f619399d16a36fc0fd7de67eac94791d8288f61c544bcd4bc7413f2a266e09d6 WatchSource:0}: Error finding container f619399d16a36fc0fd7de67eac94791d8288f61c544bcd4bc7413f2a266e09d6: Status 404 returned error can't find the container with id f619399d16a36fc0fd7de67eac94791d8288f61c544bcd4bc7413f2a266e09d6 Oct 30 22:18:46.388563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:46.388482 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" event={"ID":"6b4b704b-1b8e-48f1-a515-bc7230b8fb89","Type":"ContainerStarted","Data":"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24"} Oct 30 22:18:46.388563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:46.388523 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" event={"ID":"6b4b704b-1b8e-48f1-a515-bc7230b8fb89","Type":"ContainerStarted","Data":"f619399d16a36fc0fd7de67eac94791d8288f61c544bcd4bc7413f2a266e09d6"} Oct 30 22:18:46.502884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:46.502837 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podStartSLOduration=1.502824047 podStartE2EDuration="1.502824047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:46.500233713 +0000 UTC m=+1047.095322020" watchObservedRunningTime="2026-10-30 22:18:46.502824047 +0000 UTC m=+1047.097912372" Oct 30 22:18:46.754543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:46.754515 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:55.755665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:55.755309 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:55.776933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:55.776904 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:56.450162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.450124 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:18:56.454620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.454596 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.454921 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.454900 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:18:56.549612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.549576 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"openshift-service-ca.crt\"" Oct 30 22:18:56.549612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.549578 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"kube-root-ca.crt\"" Oct 30 22:18:56.549846 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.549580 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"default-dockercfg-kv4tv\"" Oct 30 22:18:56.549846 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.549648 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"distinct-w1-af30a9-kserve-self-signed-certs\"" Oct 30 22:18:56.552672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.552647 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx99w\" (UniqueName: \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.552814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.552676 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.552887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.552831 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.553082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.553061 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.553141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.553096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.553218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.553198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.557686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.557129 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:18:56.654340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654300 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx99w\" (UniqueName: \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654391 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654538 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654571 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.654900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.654872 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.655089 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.655066 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.655166 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.655117 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.657829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.657270 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.657829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.657474 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.722741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.722656 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx99w\" (UniqueName: \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\") pod \"distinct-w1-af30a9-kserve-84c44cd7fd-wnklc\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:56.766994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:56.766968 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:18:57.190871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:57.190844 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:18:57.197022 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:18:57.196996 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1597862_e6b8_44b2_9c59_4cf83474e109.slice/crio-306eddf452144264037645bb6c149f6e2a7e46949e8a8e443e17e365c2477cae WatchSource:0}: Error finding container 306eddf452144264037645bb6c149f6e2a7e46949e8a8e443e17e365c2477cae: Status 404 returned error can't find the container with id 306eddf452144264037645bb6c149f6e2a7e46949e8a8e443e17e365c2477cae Oct 30 22:18:57.450615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:57.450532 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" event={"ID":"b1597862-e6b8-44b2-9c59-4cf83474e109","Type":"ContainerStarted","Data":"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217"} Oct 30 22:18:57.450615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:57.450584 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" event={"ID":"b1597862-e6b8-44b2-9c59-4cf83474e109","Type":"ContainerStarted","Data":"306eddf452144264037645bb6c149f6e2a7e46949e8a8e443e17e365c2477cae"} Oct 30 22:18:57.561868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:57.561821 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podStartSLOduration=1.561790139 podStartE2EDuration="1.561790139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:57.561012717 +0000 UTC m=+1058.156101023" watchObservedRunningTime="2026-10-30 22:18:57.561790139 +0000 UTC m=+1058.156878446" Oct 30 22:18:57.767539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:18:57.767454 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:19:03.709567 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.709538 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:19:03.713646 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.713624 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.819420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.819393 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:19:03.847594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847563 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847597 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847706 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847742 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxfl\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847843 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847864 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847906 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.847948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847938 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.848022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.847977 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949093 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949130 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949254 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949310 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949344 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949466 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949522 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxfl\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949545 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949623 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949654 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949793 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949742 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.949938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949907 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.950058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.949923 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.950172 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.950150 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.951568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.951544 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:03.951960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:03.951931 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:04.008816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.008729 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:04.008933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.008851 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxfl\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:04.034910 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.034891 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:04.298847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.298768 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:19:04.303730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.303708 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.354858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.354832 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d2d4c3-openshift-default-dockercfg-kzbw6\"" Oct 30 22:19:04.370395 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.369901 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:19:04.418587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.418558 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:19:04.424132 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:19:04.424103 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb77e60_ea37_4985_8d4b_70ee10be88f8.slice/crio-8fb501732b72b9ee661bc15ba5e797770fade9766e6f7c05484f88c5a65ef56e WatchSource:0}: Error finding container 8fb501732b72b9ee661bc15ba5e797770fade9766e6f7c05484f88c5a65ef56e: Status 404 returned error can't find the container with id 8fb501732b72b9ee661bc15ba5e797770fade9766e6f7c05484f88c5a65ef56e Oct 30 22:19:04.426696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.426664 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:04.426832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.426740 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:04.426832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.426785 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:04.455992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.455969 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456009 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456033 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456138 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456170 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456203 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqntg\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456325 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456305 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456360 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456346 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.456391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.456370 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.490000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.489969 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" event={"ID":"4bb77e60-ea37-4985-8d4b-70ee10be88f8","Type":"ContainerStarted","Data":"8fb501732b72b9ee661bc15ba5e797770fade9766e6f7c05484f88c5a65ef56e"} Oct 30 22:19:04.557317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557239 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557285 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqntg\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557382 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557496 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557600 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557645 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557674 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557766 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.557898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.558012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.557950 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.558095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.558067 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.558235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.558210 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.558314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.558260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.560240 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.560214 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.560362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.560340 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.623135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.623098 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.623342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.623323 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqntg\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:04.918296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:04.918204 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:05.389746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.389710 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:19:05.419969 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:19:05.419937 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9896e29_0f85_49f1_94d9_7597e266f71c.slice/crio-a9c67d418760b46745007d3fdf95d0655f7e01b9de915b62daecb7466777ac41 WatchSource:0}: Error finding container a9c67d418760b46745007d3fdf95d0655f7e01b9de915b62daecb7466777ac41: Status 404 returned error can't find the container with id a9c67d418760b46745007d3fdf95d0655f7e01b9de915b62daecb7466777ac41 Oct 30 22:19:05.422434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.422398 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:05.422544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.422468 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:05.422544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.422513 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:05.497264 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.497226 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" event={"ID":"c9896e29-0f85-49f1-94d9-7597e266f71c","Type":"ContainerStarted","Data":"a9c67d418760b46745007d3fdf95d0655f7e01b9de915b62daecb7466777ac41"} Oct 30 22:19:05.499051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.499026 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" event={"ID":"4bb77e60-ea37-4985-8d4b-70ee10be88f8","Type":"ContainerStarted","Data":"443808b7fb8ee8a315c33fe22cda5c104f46551fe963e5ae6ed93f3f8a13fc87"} Oct 30 22:19:05.630176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:05.630122 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podStartSLOduration=2.630107057 podStartE2EDuration="2.630107057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:05.628259751 +0000 UTC m=+1066.223348059" watchObservedRunningTime="2026-10-30 22:19:05.630107057 +0000 UTC m=+1066.225195362" Oct 30 22:19:06.035443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.035411 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:06.036489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.036472 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:06.036533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.036509 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:06.507079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.507043 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" event={"ID":"c9896e29-0f85-49f1-94d9-7597e266f71c","Type":"ContainerStarted","Data":"1974ffeec86a603f5743bebef08047c013ebeb270f7f21c9fd579d45287ee8a3"} Oct 30 22:19:06.772242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.772139 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:19:06.802478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.802449 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:19:06.917281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.917212 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podStartSLOduration=2.9171892809999997 podStartE2EDuration="2.917189281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:06.67000324 +0000 UTC m=+1067.265091546" watchObservedRunningTime="2026-10-30 22:19:06.917189281 +0000 UTC m=+1067.512277589" Oct 30 22:19:06.919259 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.919240 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:06.920558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.920538 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:06.920625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:06.920581 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:07.035373 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:07.035290 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:07.035373 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:07.035348 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:07.524420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:07.524384 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:19:07.919579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:07.919495 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:07.919579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:07.919554 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:08.035522 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:08.035494 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:08.035999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:08.035925 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:08.919486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:08.919434 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:08.919689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:08.919506 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:09.035730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:09.035692 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:09.036249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:09.035757 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:09.919238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:09.919211 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:09.919456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:09.919435 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:10.035197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:10.035163 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:10.035360 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:10.035209 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:10.919305 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:10.919262 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:10.919697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:10.919329 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:11.035672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:11.035636 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 22:19:11.035845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:11.035696 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 22:19:11.919354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:11.919319 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:11.919777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:11.919371 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 22:19:12.039851 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.039793 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:12.040306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.040280 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:12.040991 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.040971 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:19:12.922874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.922844 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:12.923338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.923213 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:12.923934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:19:12.923912 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:19:16.120352 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:19:16.120309 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:19:46.109754 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:19:46.109717 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:20:16.146364 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:20:16.146322 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:20:46.109307 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:20:46.109268 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:20:47.214026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.213993 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:20:47.217775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.217755 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.267239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.267215 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"openshift-service-ca.crt\"" Oct 30 22:20:47.273560 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.273532 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th79s\" (UniqueName: \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.273848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.273823 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.273964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.273864 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.274060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.274039 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.274128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.274083 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.274233 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.274217 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.313685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.313660 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"distinct-2-w5-f80a0e-kserve-self-signed-certs\"" Oct 30 22:20:47.313685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.313677 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"default-dockercfg-b9tzf\"" Oct 30 22:20:47.313912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.313704 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"kube-root-ca.crt\"" Oct 30 22:20:47.316526 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.316499 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:20:47.394564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.394493 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.394865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.394840 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.395871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.395846 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.396426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.396408 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.397017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.396994 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.397206 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.397187 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th79s\" (UniqueName: \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.398461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.396873 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.398627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.395341 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.398838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.396330 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.413832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.413482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.414467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.414444 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.456110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.456077 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th79s\" (UniqueName: \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\") pod \"distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.530568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.530493 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:47.904433 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:47.904352 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:20:47.910168 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:20:47.910140 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a27b5a8_9b9c_462b_b022_063b34a127b0.slice/crio-2f366716be06a9c5c4bd1d839b60ad5823ae3ca3c46bc64d9f11e7a8870ade8c WatchSource:0}: Error finding container 2f366716be06a9c5c4bd1d839b60ad5823ae3ca3c46bc64d9f11e7a8870ade8c: Status 404 returned error can't find the container with id 2f366716be06a9c5c4bd1d839b60ad5823ae3ca3c46bc64d9f11e7a8870ade8c Oct 30 22:20:48.098686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:48.091285 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" event={"ID":"2a27b5a8-9b9c-462b-b022-063b34a127b0","Type":"ContainerStarted","Data":"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9"} Oct 30 22:20:48.098686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:48.091343 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" event={"ID":"2a27b5a8-9b9c-462b-b022-063b34a127b0","Type":"ContainerStarted","Data":"2f366716be06a9c5c4bd1d839b60ad5823ae3ca3c46bc64d9f11e7a8870ade8c"} Oct 30 22:20:48.202503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:48.202452 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podStartSLOduration=1.202437079 podStartE2EDuration="1.202437079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:48.20153727 +0000 UTC m=+1168.796625577" watchObservedRunningTime="2026-10-30 22:20:48.202437079 +0000 UTC m=+1168.797525384" Oct 30 22:20:48.530754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:48.530679 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:57.530814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:57.530763 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:57.543681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:57.543657 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:20:58.176817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:20:58.176776 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:21:05.490632 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.490599 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:21:05.497704 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.497660 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549092 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549060 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2nh\" (UniqueName: \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549248 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549131 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549248 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549212 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549325 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549269 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549327 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.549446 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.549431 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.553975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.553957 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"distinct-2-w1-af30a9-kserve-self-signed-certs\"" Oct 30 22:21:05.603457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.603427 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.652778 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.652850 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.652950 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.653042 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2nh\" (UniqueName: \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.653114 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.653181 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.653677 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.659834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.653954 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.674791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.674042 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.676777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.676748 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.685363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.679167 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.732012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.731983 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2nh\" (UniqueName: \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\") pod \"distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:05.813484 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:05.813402 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:06.255630 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:06.255597 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:21:06.259813 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:21:06.259769 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08457a3e_e38f_4246_babd_2542900363c7.slice/crio-e35de184b06c8fd39e9e16a53cd5c2b851e22cd5fa2c84eda108404f07f77b68 WatchSource:0}: Error finding container e35de184b06c8fd39e9e16a53cd5c2b851e22cd5fa2c84eda108404f07f77b68: Status 404 returned error can't find the container with id e35de184b06c8fd39e9e16a53cd5c2b851e22cd5fa2c84eda108404f07f77b68 Oct 30 22:21:07.262364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:07.262326 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" event={"ID":"08457a3e-e38f-4246-babd-2542900363c7","Type":"ContainerStarted","Data":"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c"} Oct 30 22:21:07.262745 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:07.262375 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" event={"ID":"08457a3e-e38f-4246-babd-2542900363c7","Type":"ContainerStarted","Data":"e35de184b06c8fd39e9e16a53cd5c2b851e22cd5fa2c84eda108404f07f77b68"} Oct 30 22:21:07.378626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:07.378581 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podStartSLOduration=2.378568768 podStartE2EDuration="2.378568768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:07.376582538 +0000 UTC m=+1187.971670843" watchObservedRunningTime="2026-10-30 22:21:07.378568768 +0000 UTC m=+1187.973657073" Oct 30 22:21:07.814015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:07.813989 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:15.813900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:15.813862 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:15.826576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:15.826553 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:16.108646 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:21:16.108562 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:21:16.354488 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:16.354456 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:19.646659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:19.646626 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:21:19.647735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:19.647714 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:21:34.811688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.811586 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:21:34.815897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.815873 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.846634 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846602 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.846773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846653 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.846773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846710 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.846912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846832 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.846912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846889 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.847014 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.846964 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87rm\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.847068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.847036 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.847068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.847061 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.847171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.847146 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.916907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.916869 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:21:34.947895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.947863 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.947975 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948020 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948053 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948127 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948155 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948253 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87rm\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948301 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948347 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.948526 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948488 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.949002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.948970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.949562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.949535 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.949837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.949791 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.950626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.950575 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.952068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.952040 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:34.952236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:34.952220 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:35.006726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.006697 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:35.007076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.007057 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87rm\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:35.132217 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.132135 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:35.569478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.569446 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:21:35.575548 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:21:35.575511 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892db7da_515e_4c64_9ede_c55394a87f03.slice/crio-7e3f2b4f5f1d563dd1f64b5f81752b5bf997d2e1d231804b28f3faae086d7595 WatchSource:0}: Error finding container 7e3f2b4f5f1d563dd1f64b5f81752b5bf997d2e1d231804b28f3faae086d7595: Status 404 returned error can't find the container with id 7e3f2b4f5f1d563dd1f64b5f81752b5bf997d2e1d231804b28f3faae086d7595 Oct 30 22:21:35.578063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.578030 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.578143 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.578098 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.578143 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:35.578134 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:36.476375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:36.476328 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" event={"ID":"892db7da-515e-4c64-9ede-c55394a87f03","Type":"ContainerStarted","Data":"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591"} Oct 30 22:21:36.476375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:36.476400 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" event={"ID":"892db7da-515e-4c64-9ede-c55394a87f03","Type":"ContainerStarted","Data":"7e3f2b4f5f1d563dd1f64b5f81752b5bf997d2e1d231804b28f3faae086d7595"} Oct 30 22:21:36.639371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:36.639317 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podStartSLOduration=2.639303538 podStartE2EDuration="2.639303538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:36.637229655 +0000 UTC m=+1217.232317974" watchObservedRunningTime="2026-10-30 22:21:36.639303538 +0000 UTC m=+1217.234391843" Oct 30 22:21:37.133309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:37.133273 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:37.134880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:37.134854 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:37.135008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:37.134901 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:38.132551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:38.132511 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:38.133036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:38.132580 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:39.132595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:39.132557 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:39.133101 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:39.132619 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:40.132647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:40.132605 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:40.133151 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:40.132671 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:41.133284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.133244 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:41.133954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.133900 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:41.166917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.166873 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.343956 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.343904 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 22:21:41.447107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.447062 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.512928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.512877 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.551351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.551318 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:21:41.551648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.551620 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.582581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.582537 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.906019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.905908 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.994349 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:41.991881 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:21:42.164877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.160924 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:42.164877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.160982 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:42.190872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.168874 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.190872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.169259 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:21:42.387191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.387076 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.388365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.387614 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:42.420287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.420191 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.483663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.483632 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:21:42.526974 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:42.526842 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:21:43.133396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:43.133363 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:21:43.133585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:43.133421 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:21:44.142122 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:44.142095 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:44.142612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:44.142591 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:44.143387 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:21:44.143369 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:21:46.109385 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:21:46.109344 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:22:04.000588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.000554 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:22:04.005082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.005061 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024116 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024170 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024238 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024262 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024299 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024429 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024475 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sb6n\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024586 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.024620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.024607 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126322 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126612 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sb6n\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126717 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126756 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126906 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.126994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.126978 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.127052 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.127023 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.127106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.127055 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.127157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.127117 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.127929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.127901 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.128112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.128083 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.128553 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.128530 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.128751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.128728 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.129313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.129002 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.130922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.130889 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.138299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.138273 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.187450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.187418 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.187604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.187533 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sb6n\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.204477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.204450 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:22:04.370877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.346664 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:04.810228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.807985 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:22:04.819682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.819651 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.852558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.852522 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:22:04.857528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.857505 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.863113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863080 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.863225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863139 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863171 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863288 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863458 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863501 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863603 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.863773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863641 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.863773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863692 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.863935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863895 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.863935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.863926 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.864053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864020 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvql4\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.864110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864085 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.864177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864114 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.864177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864145 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.864290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864175 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.864290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864200 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlhw\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:04.864401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.864293 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:04.873588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.873562 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:22:04.903868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:04.902932 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179322 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179393 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179456 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179487 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179528 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179566 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179600 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179655 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvql4\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179758 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179816 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179860 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.179889 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlhw\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.180014 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.180078 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.180131 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.180166 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.180393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.180225 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.181847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.181123 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.181847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.181467 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.184624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.184350 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:22:05.184859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.184832 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.185316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.185294 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.205286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.204403 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.205286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.204427 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.206269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.206239 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.206436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.206386 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.206965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.206926 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.207732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.207222 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.207732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.207523 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.207732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.207526 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.210219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.210195 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.210898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.210846 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.211019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.210996 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.211086 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.211071 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.211132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.211098 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.283453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.283394 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlhw\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.290442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.290373 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvql4\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.346907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.346367 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.346907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.346852 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.442371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.442093 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:05.509357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.508240 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:05.920819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:05.912486 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" event={"ID":"002d6c06-0d4a-46d2-b140-c7ad80daa567","Type":"ContainerStarted","Data":"78c7cbbe2412b0408770d457960f4e502b880bc32f4c0f350f9fdbbf5dbc5920"} Oct 30 22:22:06.261419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.261364 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:22:06.484919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.460890 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:06.557625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.557564 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:06.613003 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.593938 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:06.883892 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.859255 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:06.887658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.887619 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 22:22:06.919097 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.919054 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:22:06.963832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:06.957341 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:22:07.114245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.114197 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.114549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.114522 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.129820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.114690 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.194830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.181485 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:22:07.258822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.252482 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" event={"ID":"91068286-7de3-4d59-8d59-20180fe271c0","Type":"ContainerStarted","Data":"d3f3e8f0d4c7a1687c8f9f9365fa902524a25627db5de637f4dd1b60d7ac064c"} Oct 30 22:22:07.285158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.285118 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" event={"ID":"002d6c06-0d4a-46d2-b140-c7ad80daa567","Type":"ContainerStarted","Data":"4ae9bd24ab65f0359bb37ab8a8ecc04cead47d388deaa5f7b19f10722c182761"} Oct 30 22:22:07.331213 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:07.331175 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109521bf_2196_4621_a52c_f0330c45f94d.slice/crio-e54f0d7baef0533c4f430bcd950e1bd3c1c9b985a9d9a96f2171e6dbab8208e7 WatchSource:0}: Error finding container e54f0d7baef0533c4f430bcd950e1bd3c1c9b985a9d9a96f2171e6dbab8208e7: Status 404 returned error can't find the container with id e54f0d7baef0533c4f430bcd950e1bd3c1c9b985a9d9a96f2171e6dbab8208e7 Oct 30 22:22:07.350241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.350203 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.370954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.370907 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.371217 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.371196 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.403100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.403066 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:07.403345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.403315 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:07.440557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.440532 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:07.499971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:07.442153 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podStartSLOduration=4.442133543 podStartE2EDuration="4.442133543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:07.429841228 +0000 UTC m=+1248.024929536" watchObservedRunningTime="2026-10-30 22:22:07.442133543 +0000 UTC m=+1248.037221850" Oct 30 22:22:08.310153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.310113 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" event={"ID":"91068286-7de3-4d59-8d59-20180fe271c0","Type":"ContainerStarted","Data":"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54"} Oct 30 22:22:08.315367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.315315 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" event={"ID":"109521bf-2196-4621-a52c-f0330c45f94d","Type":"ContainerStarted","Data":"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1"} Oct 30 22:22:08.317075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.317050 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" event={"ID":"109521bf-2196-4621-a52c-f0330c45f94d","Type":"ContainerStarted","Data":"e54f0d7baef0533c4f430bcd950e1bd3c1c9b985a9d9a96f2171e6dbab8208e7"} Oct 30 22:22:08.362167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.362126 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.362422 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.362381 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:08.450826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.447118 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:08.452110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.452079 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.452382 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.452306 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:08.494439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.494321 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podStartSLOduration=4.494299167 podStartE2EDuration="4.494299167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.474654259 +0000 UTC m=+1249.069742566" watchObservedRunningTime="2026-10-30 22:22:08.494299167 +0000 UTC m=+1249.089387475" Oct 30 22:22:08.544825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.541866 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:08.544825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.543246 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.544825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.543298 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:08.605894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:08.604386 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podStartSLOduration=4.604369115 podStartE2EDuration="4.604369115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.602340697 +0000 UTC m=+1249.197429006" watchObservedRunningTime="2026-10-30 22:22:08.604369115 +0000 UTC m=+1249.199457424" Oct 30 22:22:09.368823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.358182 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.368823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.358242 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:09.442716 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.442677 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.442931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.442731 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:09.524059 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.524021 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.524325 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:09.524299 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:10.364191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.364153 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.364507 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.364477 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:10.443105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.443075 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.443502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.443471 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:10.516408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.516372 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.516672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:10.516645 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:11.358172 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.358133 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.358461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.358429 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:11.443493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.443459 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.443877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.443510 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:11.534512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.520634 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.534512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:11.532932 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:12.359237 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.359204 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.359499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.359463 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:12.442780 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.442747 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.442999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.442834 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:12.520821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.516137 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.520821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:12.516214 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:13.365821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.360112 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.365821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.360174 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 22:22:13.443249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.443212 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.443439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.443280 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:13.519818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.516735 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.519818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:13.516817 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:14.442543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.442504 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:22:14.442988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.442583 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:22:14.513414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.513380 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:22:14.513583 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.513449 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:22:14.659819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.659010 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:14.660702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.660670 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:14.663562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:14.663518 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:22:15.748881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.748839 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:15.749823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.749785 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:15.750781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.750760 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:22:15.779035 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.779010 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:15.779403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.779372 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:15.784973 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:15.784939 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:22:16.139133 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:22:16.139044 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:22:19.280251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.280217 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:22:19.284665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.284638 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.344945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.344912 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:22:19.400595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400550 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.400783 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400611 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.400783 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400645 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.400783 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400750 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkskj\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.400783 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400781 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.400993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.400962 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.401066 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.401049 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.401288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.401079 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.401288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.401110 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.501871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.501834 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.501871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.501873 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.501995 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502084 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkskj\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502107 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502228 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502315 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502299 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502327 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502356 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502500 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502644 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502675 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502726 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.502958 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.502931 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.504206 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.504185 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.504604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.504581 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.564468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.564384 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.564644 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.564618 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkskj\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.582729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.582695 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:22:19.588573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.588546 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.599837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.599809 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:19.680774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.673773 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706079 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706188 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706231 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706293 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706349 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706392 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706691 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706732 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.707135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.706872 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwts\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.813767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.813726 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwts\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814032 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814122 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814163 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814231 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814292 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814338 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814387 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814417 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814775 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.814844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814781 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.815047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.814985 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.815498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.815469 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.815605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.815550 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.817906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.817884 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.818568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.818547 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.871719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.871688 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.871872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.871697 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwts\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:19.903874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:19.903835 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:20.051651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.051618 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:22:20.054646 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:20.054604 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65fccc6_fd3f_4757_b5bb_bd2254834e73.slice/crio-b500ae3bef1f79e2a8a11aafa46d58061c82c43923bee352bedc0e1a7a09d752 WatchSource:0}: Error finding container b500ae3bef1f79e2a8a11aafa46d58061c82c43923bee352bedc0e1a7a09d752: Status 404 returned error can't find the container with id b500ae3bef1f79e2a8a11aafa46d58061c82c43923bee352bedc0e1a7a09d752 Oct 30 22:22:20.057572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.057538 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.057660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.057627 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.057700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.057661 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.302477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.302430 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:22:20.307224 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:20.307192 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90349c34_9110_46f4_9cb1_a2a6f37b8f33.slice/crio-c93f78e710b8e5ce5911c06662c6936bc9cae96a1749a43bda5515d7f0a17cea WatchSource:0}: Error finding container c93f78e710b8e5ce5911c06662c6936bc9cae96a1749a43bda5515d7f0a17cea: Status 404 returned error can't find the container with id c93f78e710b8e5ce5911c06662c6936bc9cae96a1749a43bda5515d7f0a17cea Oct 30 22:22:20.310258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.310217 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.310346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.310303 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.310346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.310340 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.493547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.493278 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" event={"ID":"90349c34-9110-46f4-9cb1-a2a6f37b8f33","Type":"ContainerStarted","Data":"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26"} Oct 30 22:22:20.493547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.493350 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" event={"ID":"90349c34-9110-46f4-9cb1-a2a6f37b8f33","Type":"ContainerStarted","Data":"c93f78e710b8e5ce5911c06662c6936bc9cae96a1749a43bda5515d7f0a17cea"} Oct 30 22:22:20.496098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.496041 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" event={"ID":"e65fccc6-fd3f-4757-b5bb-bd2254834e73","Type":"ContainerStarted","Data":"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776"} Oct 30 22:22:20.496285 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.496128 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" event={"ID":"e65fccc6-fd3f-4757-b5bb-bd2254834e73","Type":"ContainerStarted","Data":"b500ae3bef1f79e2a8a11aafa46d58061c82c43923bee352bedc0e1a7a09d752"} Oct 30 22:22:20.616875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.610679 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:20.625094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.625043 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podStartSLOduration=1.625026276 podStartE2EDuration="1.625026276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:20.617307736 +0000 UTC m=+1261.212396065" watchObservedRunningTime="2026-10-30 22:22:20.625026276 +0000 UTC m=+1261.220114583" Oct 30 22:22:20.780820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.774377 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podStartSLOduration=1.774354092 podStartE2EDuration="1.774354092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:20.756963152 +0000 UTC m=+1261.352051486" watchObservedRunningTime="2026-10-30 22:22:20.774354092 +0000 UTC m=+1261.369442400" Oct 30 22:22:20.904548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:20.904431 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:21.601832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:21.601754 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.601832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:21.601823 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:21.906265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:21.906177 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.906265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:21.906229 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:22.601205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:22.600158 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.601205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:22.600210 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:22.905309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:22.905217 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.905756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:22.905296 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:23.600207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:23.600172 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:23.600461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:23.600433 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:23.904917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:23.904823 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:23.904917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:23.904903 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:24.600816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:24.600611 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:24.600816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:24.600682 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:24.904946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:24.904370 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:24.904946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:24.904424 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:25.603998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:25.603941 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:25.604681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:25.604622 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:25.908877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:25.907955 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:25.908877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:25.908017 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:26.600639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:26.600598 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:26.600835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:26.600670 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:22:26.904830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:26.904718 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:26.904830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:26.904784 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:22:27.194721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.194677 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:22:27.194999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.194955 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" containerID="cri-o://2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b" gracePeriod=30 Oct 30 22:22:27.606498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.606413 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:27.606849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.606826 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:27.607740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.607716 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:22:27.941318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.941291 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:27.942286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.942259 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:27.943592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:27.943575 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:22:32.537581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.537557 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:22:32.578575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.578548 4305 generic.go:356] "Generic (PLEG): container finished" podID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerID="2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b" exitCode=0 Oct 30 22:22:32.578674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.578596 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" Oct 30 22:22:32.578674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.578630 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" event={"ID":"863ba88e-0df6-4782-bcd4-1ca3813b3d0a","Type":"ContainerDied","Data":"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b"} Oct 30 22:22:32.578757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.578684 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh" event={"ID":"863ba88e-0df6-4782-bcd4-1ca3813b3d0a","Type":"ContainerDied","Data":"a21b5bfd980ded17b6c98f9b35d5a59dfd6034ed4d4121d7691d02b98fe2eee9"} Oct 30 22:22:32.578757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.578714 4305 scope.go:122] "RemoveContainer" containerID="2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b" Oct 30 22:22:32.588022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.588006 4305 scope.go:122] "RemoveContainer" containerID="2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b" Oct 30 22:22:32.588306 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:22:32.588289 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b\": container with ID starting with 2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b not found: ID does not exist" containerID="2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b" Oct 30 22:22:32.588392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.588314 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b"} err="failed to get container status \"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b\": rpc error: code = NotFound desc = could not find container \"2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b\": container with ID starting with 2346a35a8a51c51f00e664d40102b479dcd4c9995cfb2ab59ddcd96798c0304b not found: ID does not exist" Oct 30 22:22:32.640724 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640702 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.640822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640743 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.640822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640778 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.640895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640836 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.640895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640879 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.641047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640945 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.641047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.640973 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.641047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641045 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.641487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641061 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:32.641487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641080 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\") pod \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\" (UID: \"863ba88e-0df6-4782-bcd4-1ca3813b3d0a\") " Oct 30 22:22:32.641487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641132 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:32.641487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641165 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:32.641487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641433 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:32.641821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641717 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:32.641893 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641874 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.641987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641900 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.641987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641915 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.641987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641929 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.641987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.641943 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.643391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.643356 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:22:32.643391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.643365 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:32.643512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.643383 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "kube-api-access-mmgvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:32.643512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.643497 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo" pod "863ba88e-0df6-4782-bcd4-1ca3813b3d0a" (UID: "863ba88e-0df6-4782-bcd4-1ca3813b3d0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:22:32.743456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.743424 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgvl\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-kube-api-access-mmgvl\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.743456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.743450 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.743456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.743459 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.743664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.743470 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863ba88e-0df6-4782-bcd4-1ca3813b3d0a-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:22:32.994470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.994432 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:22:33.000015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:32.999988 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-clhnh"] Oct 30 22:22:33.519988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:33.519956 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" path="/var/lib/kubelet/pods/863ba88e-0df6-4782-bcd4-1ca3813b3d0a/volumes" Oct 30 22:22:35.606591 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.606556 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:22:35.607398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.607132 4305 cpu_manager.go:414] "containerMap: removing container" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" Oct 30 22:22:35.607398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.607145 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" Oct 30 22:22:35.607398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.607284 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="863ba88e-0df6-4782-bcd4-1ca3813b3d0a" containerName="istio-proxy" Oct 30 22:22:35.612241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.612223 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.681834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.681781 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.681898 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.681935 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.681963 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.682067 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.682091 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zk9z\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.682127 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682253 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.682142 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.682287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.682269 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783356 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783415 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783436 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783462 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783504 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783530 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zk9z\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783572 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783591 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.783929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.783851 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.784205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.784181 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.784301 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.784221 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.784354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.784336 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.784431 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.784411 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.786068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.786041 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.786200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.786185 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.814149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.814106 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:22:35.851007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.850814 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zk9z\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.851007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.850936 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.886791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.886708 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:22:35.892211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.892188 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.926272 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.926243 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:35.953601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.953487 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:22:35.987566 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.987521 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.987763 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.987738 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.987944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.987881 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988023 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988006 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhwd\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988267 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988225 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988293 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988365 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988393 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:35.988505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:35.988428 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.089969 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.089929 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.089978 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090010 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090121 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090171 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090228 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090369 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090382 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090449 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090483 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhwd\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090662 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090636 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090723 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090666 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.090964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.090938 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.091077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.091051 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.092472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.092453 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.092904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.092887 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.147508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.147429 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhwd\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.147508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.147482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.207669 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.207634 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:36.378368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.378337 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:22:36.382892 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:36.382865 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427438a0_38f5_431c_a210_25230914a082.slice/crio-94c3ebc1e8a7e7d8ce545d30905881fe838801657c9282b700f62ba512a3f260 WatchSource:0}: Error finding container 94c3ebc1e8a7e7d8ce545d30905881fe838801657c9282b700f62ba512a3f260: Status 404 returned error can't find the container with id 94c3ebc1e8a7e7d8ce545d30905881fe838801657c9282b700f62ba512a3f260 Oct 30 22:22:36.385359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.385316 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.385463 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.385435 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.385531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.385488 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.592294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.592249 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:22:36.597517 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:36.597479 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7d9091_a50e_4d27_af57_e098d735a24e.slice/crio-1f0f005c4b7840dc585a431bf6cca37ced95e76ab08339d0dffabdcaad207feb WatchSource:0}: Error finding container 1f0f005c4b7840dc585a431bf6cca37ced95e76ab08339d0dffabdcaad207feb: Status 404 returned error can't find the container with id 1f0f005c4b7840dc585a431bf6cca37ced95e76ab08339d0dffabdcaad207feb Oct 30 22:22:36.600056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.600026 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.600147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.600104 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.600191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.600171 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.606126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.606100 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" event={"ID":"9f7d9091-a50e-4d27-af57-e098d735a24e","Type":"ContainerStarted","Data":"1f0f005c4b7840dc585a431bf6cca37ced95e76ab08339d0dffabdcaad207feb"} Oct 30 22:22:36.608104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.608077 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" event={"ID":"427438a0-38f5-431c-a210-25230914a082","Type":"ContainerStarted","Data":"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120"} Oct 30 22:22:36.608492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.608123 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" event={"ID":"427438a0-38f5-431c-a210-25230914a082","Type":"ContainerStarted","Data":"94c3ebc1e8a7e7d8ce545d30905881fe838801657c9282b700f62ba512a3f260"} Oct 30 22:22:36.723071 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.723007 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podStartSLOduration=1.7229868929999999 podStartE2EDuration="1.722986893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:36.721481047 +0000 UTC m=+1277.316569366" watchObservedRunningTime="2026-10-30 22:22:36.722986893 +0000 UTC m=+1277.318075193" Oct 30 22:22:36.927115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:36.927030 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:37.625440 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:37.625411 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" event={"ID":"9f7d9091-a50e-4d27-af57-e098d735a24e","Type":"ContainerStarted","Data":"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae"} Oct 30 22:22:37.808532 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:37.808478 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podStartSLOduration=2.8084628130000002 podStartE2EDuration="2.808462813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.807141891 +0000 UTC m=+1278.402230199" watchObservedRunningTime="2026-10-30 22:22:37.808462813 +0000 UTC m=+1278.403551532" Oct 30 22:22:37.928719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:37.928634 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:37.928719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:37.928690 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:38.207852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:38.207788 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:38.208934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:38.208909 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:38.209006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:38.208958 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:38.927054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:38.927023 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:38.927475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:38.927072 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:39.208719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:39.208681 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:39.208924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:39.208746 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:39.926914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:39.926883 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:39.927076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:39.926931 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:40.208082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:40.208050 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:40.208247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:40.208104 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:40.927369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:40.927328 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:40.927763 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:40.927386 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:41.208811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:41.208756 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:41.208993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:41.208819 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:41.962905 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:41.953938 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:41.962905 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:41.954018 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:42.207982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:42.207951 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:42.208154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:42.208001 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:42.927500 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:42.927464 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:42.927672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:42.927514 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:22:43.208242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:43.208209 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:43.208633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:43.208277 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:22:44.046603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.046561 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:44.048145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.048122 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:44.049606 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.049578 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:22:44.484559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.484528 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:44.485232 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.485175 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:44.486364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:44.486344 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:22:46.110258 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:22:46.110184 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:22:48.489934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:48.489885 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:48.541976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:48.541870 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:48.916986 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:48.916883 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:48.977492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:48.977445 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.264819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.263939 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:22:49.289821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.280925 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.525880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.517873 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:22:49.525880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.518293 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.525880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.518863 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:22:49.600821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.583413 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:22:49.700822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.700212 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.934131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.934035 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:22:49.999010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.998981 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:22:50.000061 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:49.997057 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.000812 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.000776 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:22:50.334244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.334202 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.393460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.336129 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:22:50.393859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.393833 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:22:50.415554 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.415526 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:22:50.477440 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.477392 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:22:50.536349 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.536308 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.553145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.541321 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.553145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.541404 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.553145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.552348 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:22:50.639888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.628352 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.639888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.628623 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:22:50.881936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:50.881891 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.024829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.012339 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.024829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.013236 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.054019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.031003 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:22:51.522823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.511187 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:22:51.522823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.511937 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:22:51.554450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.554412 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 22:22:51.555306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.555279 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:22:51.623984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.601689 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:22:51.800884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.624762 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.918725 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:51.912190 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.094877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.082144 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:22:52.192912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.133875 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:22:52.192912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.134151 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:22:52.201065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.201037 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:22:52.241519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.241496 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:22:52.241983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.213198 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:22:52.288176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.255940 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:22:52.288422 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.288407 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:22:52.288710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.279452 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.288884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.288762 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.288884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.288850 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.288930 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.288979 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.289019 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8k7\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.289219 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.289323 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.289474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.289456 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.314029 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.312973 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:22:52.390441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390349 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390446 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390478 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8k7\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390708 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.390952 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.390934 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.391047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.391032 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.391104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.391079 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.392042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.392017 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.393029 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.393000 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.393455 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.393432 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.394109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.394084 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.394488 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.394465 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.408080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.408050 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.408692 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.408663 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.456847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.456810 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8k7\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.463146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.462784 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.495901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.495854 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:52.952442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.952413 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:22:52.957829 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:22:52.957786 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ea0083_1fd5_4023_8c74_9d2978145014.slice/crio-ff9090fc81ec63bd17ea784d8c98569f124e05275ac26d73d017cee28f77db7a WatchSource:0}: Error finding container ff9090fc81ec63bd17ea784d8c98569f124e05275ac26d73d017cee28f77db7a: Status 404 returned error can't find the container with id ff9090fc81ec63bd17ea784d8c98569f124e05275ac26d73d017cee28f77db7a Oct 30 22:22:52.960716 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.960672 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:52.960826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.960757 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:52.960884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:52.960846 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:53.824539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:53.824505 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" event={"ID":"c4ea0083-1fd5-4023-8c74-9d2978145014","Type":"ContainerStarted","Data":"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f"} Oct 30 22:22:53.824741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:53.824558 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" event={"ID":"c4ea0083-1fd5-4023-8c74-9d2978145014","Type":"ContainerStarted","Data":"ff9090fc81ec63bd17ea784d8c98569f124e05275ac26d73d017cee28f77db7a"} Oct 30 22:22:53.939407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:53.939356 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podStartSLOduration=3.939341392 podStartE2EDuration="3.939341392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:53.938334298 +0000 UTC m=+1294.533422607" watchObservedRunningTime="2026-10-30 22:22:53.939341392 +0000 UTC m=+1294.534429698" Oct 30 22:22:54.497273 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:54.497236 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:22:54.498414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:54.498385 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:54.498551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:54.498442 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:22:55.496501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:55.496469 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:55.496653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:55.496531 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:22:56.496598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:56.496560 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:56.497026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:56.496614 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:22:57.496919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:57.496886 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:57.497299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:57.496937 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:22:58.497417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:58.497373 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:58.497950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:58.497440 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:22:59.497265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:59.497230 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:59.497444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:22:59.497290 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:23:00.500890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:00.500862 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:23:00.501344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:00.501318 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:23:00.502072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:00.502053 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:23:16.109868 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:23:16.109829 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:23:19.779017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:19.778980 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:23:19.785564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:19.785539 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:19.849813 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:19.849362 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dee83803-openshift-default-dockercfg-4hd5s\"" Oct 30 22:23:19.993818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:19.857551 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:23:20.003069 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:19.897902 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158190 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddf2\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158281 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158326 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158379 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158496 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158533 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158629 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.159407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.158709 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.217512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.217481 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:23:20.288914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.217851 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.290476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.290448 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:23:20.290688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.290673 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:23:20.304874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.304847 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:23:20.305324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.305293 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.327209 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.327163 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.330510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.330465 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddf2\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.330751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.330717 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.330920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.330903 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.331083 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.331067 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.331299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.331284 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.331442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.331428 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.331659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.331641 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.331862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.331846 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.360853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.358318 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.413826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.413066 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.447823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.447260 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:23:20.475757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.468161 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.475757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.470978 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.475757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.471540 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.487825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.482720 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.491818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.491652 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.495652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.495619 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddf2\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.547084 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.547001 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:23:20.550026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.549993 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\") pod \"e2e-ait-dee83803-openshift-default-54775796bf-8vttp\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.550178 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.500769 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570481 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570452 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.570588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570516 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570628 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570732 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570764 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570786 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.570936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570918 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.570993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570942 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.570993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570972 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.571128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.570999 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.571128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.571128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571110 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.571354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571229 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.571354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571282 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhl98\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.571676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571343 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.571676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571670 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.571832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571709 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trcbp\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.571832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571733 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.571832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571751 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.571832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571768 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.571832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571785 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl626\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.572095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.571959 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.572154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.572112 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.572209 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.572194 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.572285 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.572267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.572343 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.572306 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.572458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.572439 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.673787 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673754 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.673787 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673817 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673836 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhl98\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673868 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673884 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673910 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trcbp\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.673945 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674053 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674097 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674131 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl626\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674167 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674185 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674211 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674302 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674347 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674390 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674452 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674551 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674584 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674617 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674664 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.674738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674764 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674790 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674844 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674864 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674881 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674950 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.675002 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.675058 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.675064 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.674896 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.675353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.675142 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.676164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.675868 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.676164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.676060 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.676289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.676260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.676349 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.676281 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.676642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.676529 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.676642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.676585 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.677250 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.677223 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.678119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678060 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.678222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678176 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.678287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678233 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.678287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678239 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.678548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678530 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.678784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.678763 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.740792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.740763 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:20.741615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.741537 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.745641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.745416 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.745683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.745635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trcbp\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:20.750250 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.750217 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl626\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.750992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.750970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.759049 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.759030 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhl98\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.870218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.870186 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:20.879659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.879628 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:20.886100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:20.886054 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:21.200316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.200289 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:23:21.202152 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:23:21.202123 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab096bfd_3e6e_4abf_bfdb_0bdb90334a3d.slice/crio-d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965 WatchSource:0}: Error finding container d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965: Status 404 returned error can't find the container with id d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965 Oct 30 22:23:21.204978 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.204947 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.205060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.205013 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.205060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.205039 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.288141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.288113 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:23:21.292413 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:23:21.292385 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f20bbd8_30b8_419c_8534_4c83e76665be.slice/crio-6415dbe8040902c793031230e2ef3db589cd260db1d00d19b00f37db6e98145b WatchSource:0}: Error finding container 6415dbe8040902c793031230e2ef3db589cd260db1d00d19b00f37db6e98145b: Status 404 returned error can't find the container with id 6415dbe8040902c793031230e2ef3db589cd260db1d00d19b00f37db6e98145b Oct 30 22:23:21.294962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.294927 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.295073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.295007 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.295073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.295037 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.334330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.332964 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:23:21.335398 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:23:21.335370 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e539ea_bd5d_4ab6_9b2f_8d93851fec5b.slice/crio-17562b4e1694215cf0b7a94e9ef53881e19f60dc7ca46b1823ebc503d817e715 WatchSource:0}: Error finding container 17562b4e1694215cf0b7a94e9ef53881e19f60dc7ca46b1823ebc503d817e715: Status 404 returned error can't find the container with id 17562b4e1694215cf0b7a94e9ef53881e19f60dc7ca46b1823ebc503d817e715 Oct 30 22:23:21.338736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.338707 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.338854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.338768 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.338896 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.338875 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.344140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.343466 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:23:21.348551 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:23:21.348521 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be5ad7d_52cb_4a46_8d9c_51f4334a577f.slice/crio-873db526451a87e239847ea4ba68e8ff65f8fd769fa7f7a6be8d9814455aecdc WatchSource:0}: Error finding container 873db526451a87e239847ea4ba68e8ff65f8fd769fa7f7a6be8d9814455aecdc: Status 404 returned error can't find the container with id 873db526451a87e239847ea4ba68e8ff65f8fd769fa7f7a6be8d9814455aecdc Oct 30 22:23:21.351105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.351081 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.351228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.351145 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.351228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:21.351177 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:22.212819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.212756 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" event={"ID":"2be5ad7d-52cb-4a46-8d9c-51f4334a577f","Type":"ContainerStarted","Data":"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e"} Oct 30 22:23:22.213486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.213440 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" event={"ID":"2be5ad7d-52cb-4a46-8d9c-51f4334a577f","Type":"ContainerStarted","Data":"873db526451a87e239847ea4ba68e8ff65f8fd769fa7f7a6be8d9814455aecdc"} Oct 30 22:23:22.219162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.218870 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerStarted","Data":"3c118426b08b5398213f88ba52eeb708521fb6e0b0e97a22756f74775daae4dd"} Oct 30 22:23:22.219162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.218955 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerStarted","Data":"d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965"} Oct 30 22:23:22.222213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.221990 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" event={"ID":"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b","Type":"ContainerStarted","Data":"e87e41739019f271082b8cff313e41466ed29841bc5f7d7c64f6e0342bbd47b6"} Oct 30 22:23:22.222213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.222044 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" event={"ID":"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b","Type":"ContainerStarted","Data":"17562b4e1694215cf0b7a94e9ef53881e19f60dc7ca46b1823ebc503d817e715"} Oct 30 22:23:22.223184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.223159 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" event={"ID":"7f20bbd8-30b8-419c-8534-4c83e76665be","Type":"ContainerStarted","Data":"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd"} Oct 30 22:23:22.223356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.223342 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" event={"ID":"7f20bbd8-30b8-419c-8534-4c83e76665be","Type":"ContainerStarted","Data":"6415dbe8040902c793031230e2ef3db589cd260db1d00d19b00f37db6e98145b"} Oct 30 22:23:22.372843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.357711 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podStartSLOduration=2.357686969 podStartE2EDuration="2.357686969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.329995917 +0000 UTC m=+1322.925084223" watchObservedRunningTime="2026-10-30 22:23:22.357686969 +0000 UTC m=+1322.952775277" Oct 30 22:23:22.468870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.465388 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podStartSLOduration=3.4653699700000002 podStartE2EDuration="3.46536997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.463395954 +0000 UTC m=+1323.058484262" watchObservedRunningTime="2026-10-30 22:23:22.46536997 +0000 UTC m=+1323.060458277" Oct 30 22:23:22.754869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.746440 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:22.755127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.748574 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podStartSLOduration=2.748557319 podStartE2EDuration="2.748557319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.630753426 +0000 UTC m=+1323.225841732" watchObservedRunningTime="2026-10-30 22:23:22.748557319 +0000 UTC m=+1323.343645626" Oct 30 22:23:22.785864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.768863 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:22.843847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.830278 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podStartSLOduration=3.830258052 podStartE2EDuration="3.830258052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:22.745526302 +0000 UTC m=+1323.340614621" watchObservedRunningTime="2026-10-30 22:23:22.830258052 +0000 UTC m=+1323.425346362" Oct 30 22:23:22.872186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.872152 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:22.875697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.875664 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.876142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.876110 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:22.894896 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.894860 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:22.895241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.895219 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.895379 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.895355 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:22.910408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.910372 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:22.916357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.916321 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.916613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:22.916588 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:23.235153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.235122 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:23.236311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.236284 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:23:23.904871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.887027 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.904871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.887109 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:23.917988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.917947 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.918195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.918031 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:23.918195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.918130 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.918195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:23.918156 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:24.872239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.872200 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.873308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.872866 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:24.882166 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.882139 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.882366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.882340 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:24.891188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.891162 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.891389 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:24.891362 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:25.920464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:25.914910 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:25.920464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:25.914991 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:26.039342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.036902 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.039342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.036973 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:26.040044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.036662 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.040243 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.040220 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:26.929824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.925173 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.930496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.927151 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.930496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.930413 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:26.930496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.929213 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.930496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.930463 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:26.931208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:26.931179 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:27.788297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.787982 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.892501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.892462 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.892774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.892745 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:27.923823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.918967 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.923823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.919045 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:27.923823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.919136 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.923823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:27.919162 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:28.069881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.063068 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:23:28.101264 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.065291 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:28.224531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.224480 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.342998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.342896 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.542937 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.542890 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.546925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.546885 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:28.588943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.588892 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.877823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.877144 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.877823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.877205 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:28.900821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.893158 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.900821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.893216 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:28.900821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.893487 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.900821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:28.893524 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:29.157872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.152972 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.157872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.154044 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.157872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.155261 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:23:29.157872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.155515 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:23:29.234822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.209252 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:23:29.368820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.331879 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.389042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.361213 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.423872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.361343 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:29.423872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.390261 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:23:29.423872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.390429 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:23:29.423872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.361560 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 22:23:29.595616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.592022 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.595616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.593858 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.714874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.712914 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.996745 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.996656 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:29.997003 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.996969 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:30.015305 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:29.998557 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.015599 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.015552 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:30.033877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.029901 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.142177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.141870 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.142177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.141932 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:30.189826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.189140 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.217844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.191780 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.280875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.257117 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.453899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.260637 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:23:30.467981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.467945 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:23:30.468218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.468197 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:23:30.469438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.389877 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.471150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.391860 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:30.499823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.391891 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.808861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.627463 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.855819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.830537 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:23:30.910825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.627497 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:30.965962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.965905 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.966235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.966205 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:23:30.967419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.966562 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.967611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.967586 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:30.977732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.977701 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.978013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:30.977970 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:31.049052 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.049007 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 22:23:31.066909 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.066244 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.260116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.260072 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.280939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.280897 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:31.302154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.129788 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:23:31.500818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.497095 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.500818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.499141 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.500818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.500022 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:23:31.806873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.804634 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.843819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.821560 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.888869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.888830 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:31.889175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.889118 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 22:23:31.956821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:31.949053 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:32.106090 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.105964 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.128199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.128164 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:32.131843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.128451 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:23:32.265922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.265880 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.294972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.294924 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.504038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.503984 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.521255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.518888 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:32.521909 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.518958 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:23:32.813050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.812950 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:23:32.870197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.870148 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.952392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.952356 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:32.952659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.952629 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.964531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:32.964399 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:33.204882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.165220 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:23:33.316823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.282938 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.382824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.319339 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:33.598873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.590415 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.598873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.592020 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.598873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.592068 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.631823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.613314 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.877872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.877673 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:23:33.888397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.883061 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 22:23:33.969822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.960949 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:33.969822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.961017 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.969822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.961628 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:33.969822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.961694 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.969822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.967872 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.001822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.997960 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:34.001822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:33.998026 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.309928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.309879 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:23:34.377210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.377164 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:23:34.386271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.386217 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.594016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.593892 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.595111 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.595077 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.595580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.595292 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.884653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.880064 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:23:34.919411 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.917309 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 22:23:34.969680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.969634 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.003823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:34.978725 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.004141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.004100 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.034254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.033880 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.034254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.033940 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.037644 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.037612 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.037982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.037865 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.411822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.397621 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:23:35.411822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.399722 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.456702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.455660 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.527839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.526483 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:23:35.754944 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.754872 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:35.756002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.755841 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:23:35.757283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.757253 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:35.802850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.801586 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:35.828163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.812037 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:35.920822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.896956 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:35.923953 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.923924 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:35.935827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.933996 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.086260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.986014 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:36.087292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:35.988639 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:23:36.093904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.083636 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:36.093904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.092325 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.400153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.399515 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:36.400153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.399578 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.497939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.497884 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.634552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.507865 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:23:36.634552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.586173 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:23:36.668900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.656737 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:23:36.758694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.755825 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.758694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.756791 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.760441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:36.760408 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:37.134822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.110324 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 22:23:37.134822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.111240 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 22:23:37.247833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.241008 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:37.756821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.756777 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:37.768611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.760985 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:37.792274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.331626 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:37.792753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.469071 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:37.792980 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.792926 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:37.793369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.392128 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:23:37.822075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.696746 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:23:37.829579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.829524 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:23:37.900205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:37.756691 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:38.093976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.078689 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:38.112953 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.112916 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:38.221380 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.221347 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:23:38.221647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.221628 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:23:38.221847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.221833 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:38.222007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.221994 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:23:38.222129 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.222118 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:23:38.226047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.226026 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:23:38.376868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.365226 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:23:38.399593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.399561 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:38.417010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.415900 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:23:38.468207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:23:38.468026 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:23:46.188063 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:23:46.188004 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:24:00.802234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.802203 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:24:00.807850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.807830 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:00.856784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.856756 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-05e3625f-serving-cert\"" Oct 30 22:24:00.906534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.906493 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:24:00.939925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.939888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:00.940091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.939940 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:00.940091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.940068 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:00.940319 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:00.940290 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqxc\" (UniqueName: \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.041529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.041494 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqxc\" (UniqueName: \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.041681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.041623 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.041681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.041658 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.041753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.041707 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.042328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.042307 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.044175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.044144 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.044337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.044313 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.099160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.099098 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqxc\" (UniqueName: \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\") pod \"maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.124082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.124058 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:01.525380 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.525348 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:24:01.530419 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:24:01.530394 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67fafa8_7134_4fa4_bab4_51ccf9672d94.slice/crio-75799a9f6dd5111e8826165f100104faecb5c125b4b89eb115772b77702233cc WatchSource:0}: Error finding container 75799a9f6dd5111e8826165f100104faecb5c125b4b89eb115772b77702233cc: Status 404 returned error can't find the container with id 75799a9f6dd5111e8826165f100104faecb5c125b4b89eb115772b77702233cc Oct 30 22:24:01.531737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.531716 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:24:01.839668 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:01.839586 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" event={"ID":"c67fafa8-7134-4fa4-bab4-51ccf9672d94","Type":"ContainerStarted","Data":"75799a9f6dd5111e8826165f100104faecb5c125b4b89eb115772b77702233cc"} Oct 30 22:24:02.861159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:02.861089 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" event={"ID":"c67fafa8-7134-4fa4-bab4-51ccf9672d94","Type":"ContainerStarted","Data":"1f9e63ec3f8bb6c43afe43f6286bcd84e7c26a4dfdfa721c7a6edd4a1179e099"} Oct 30 22:24:02.871399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:02.867059 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:02.983395 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:02.983154 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" podStartSLOduration=2.09014607 podStartE2EDuration="2.983140175s" totalImagesPullingTime="892.994105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:02.979263259 +0000 UTC m=+1363.574351565" watchObservedRunningTime="2026-10-30 22:24:02.983140175 +0000 UTC m=+1363.578228480" Oct 30 22:24:05.127937 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.127899 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:24:05.132020 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.132002 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142237 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142214 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142255 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142389 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142366 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142404 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142547 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142625 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142718 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.142781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.142770 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ncb\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.234679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.234653 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:24:05.243824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.243778 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.243961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.243870 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.244028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.243963 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.244028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.243994 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.244519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.244458 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.245285 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.245260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.245660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.245638 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.245862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.245842 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.245991 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246236 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246614 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246589 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246589 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246678 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246728 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ncb\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.246916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.246780 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.247327 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.247301 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.304247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.304222 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.304474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.304450 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ncb\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.446970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.446935 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:05.911434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.911394 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:05.911528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.911482 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:05.911573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.911525 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:05.914452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:05.914429 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:24:06.602453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.602419 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:24:06.606727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.606705 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.676516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.676481 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.676711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.676545 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.676780 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.676731 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.676860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.676808 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.676981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.676964 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.677043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.677002 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd4s\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.677081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.677062 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.677129 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.677114 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.677188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.677161 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.710195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.710166 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:24:06.778768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.778726 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.778777 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd4s\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.778858 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.778913 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.778963 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779105 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779280 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779296 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779365 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779841 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.779906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.779863 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.780213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.780161 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.780436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.780408 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.781972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.781923 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.784577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.784550 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.836623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.836602 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.836789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.836771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd4s\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:06.918270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.918177 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" event={"ID":"a9b78c95-1ae0-4e46-83f7-7217547a25ed","Type":"ContainerStarted","Data":"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc"} Oct 30 22:24:06.918270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.918239 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" event={"ID":"a9b78c95-1ae0-4e46-83f7-7217547a25ed","Type":"ContainerStarted","Data":"dac2bc89911bb36921eeba4eb0481b0e0c1fdb0176064bf52d36c8a645f0755f"} Oct 30 22:24:06.921607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:06.921585 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:07.376004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.375936 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podStartSLOduration=2.375920464 podStartE2EDuration="2.375920464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:07.042899718 +0000 UTC m=+1367.637988050" watchObservedRunningTime="2026-10-30 22:24:07.375920464 +0000 UTC m=+1367.971008775" Oct 30 22:24:07.378523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.378498 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:24:07.381380 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:24:07.381353 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135ec54c_ac83_45ca_ae90_8f1e90855cc2.slice/crio-675b897f9420a967dc80308977aed9333b25599f5d107852cb9bd5ddda74b80b WatchSource:0}: Error finding container 675b897f9420a967dc80308977aed9333b25599f5d107852cb9bd5ddda74b80b: Status 404 returned error can't find the container with id 675b897f9420a967dc80308977aed9333b25599f5d107852cb9bd5ddda74b80b Oct 30 22:24:07.384421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.384389 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:07.384525 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.384476 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:07.384583 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.384520 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:07.458875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.458445 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:07.458875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.458560 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:07.458875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.458603 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:07.929357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.929307 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" event={"ID":"135ec54c-ac83-45ca-ae90-8f1e90855cc2","Type":"ContainerStarted","Data":"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d"} Oct 30 22:24:07.929734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:07.929387 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" event={"ID":"135ec54c-ac83-45ca-ae90-8f1e90855cc2","Type":"ContainerStarted","Data":"675b897f9420a967dc80308977aed9333b25599f5d107852cb9bd5ddda74b80b"} Oct 30 22:24:08.068823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.066727 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podStartSLOduration=2.066702358 podStartE2EDuration="2.066702358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:08.050316809 +0000 UTC m=+1368.645405126" watchObservedRunningTime="2026-10-30 22:24:08.066702358 +0000 UTC m=+1368.661790691" Oct 30 22:24:08.454825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.454347 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:08.454825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.454444 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:08.906941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.906018 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:24:08.930823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.923666 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:08.930823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.923863 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:08.930823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:08.923912 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:09.570823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.543140 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:09.570823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.543241 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:09.570823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.544429 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:24:09.570823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.549883 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:09.631673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.631619 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:09.680946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.680879 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:09.947882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.938606 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:09.947882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:09.938663 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:10.448819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.447217 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:10.448819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.447287 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:10.776875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.770139 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:24:10.833058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.822740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:10.833058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.824638 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:24:10.905202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.900075 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae20a62d-serving-cert\"" Oct 30 22:24:10.964523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.940958 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:10.964523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.941024 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:10.964523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.955903 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:10.964523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:10.955955 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.074920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.067624 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.074920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.067737 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvqz\" (UniqueName: \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.478481 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.478426 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:11.478946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.478827 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:11.510249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.510209 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.510524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.510502 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvqz\" (UniqueName: \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.513218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.512939 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.517864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.510790 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.518069 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.518044 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.609816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.543746 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:11.609816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.547449 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:24:11.609816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.550094 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.647822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.641651 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvqz\" (UniqueName: \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.768455 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.768372 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\") pod \"maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:11.924484 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.924446 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:11.924769 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.924735 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:11.929929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.929891 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:24:11.975661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:11.975627 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:12.469681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:12.469641 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:12.470001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:12.469967 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:12.534861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:12.534678 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:24:12.922277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:12.922187 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:12.922277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:12.922251 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:13.456492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.456459 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:13.457040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.457008 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:13.550822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.545266 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" event={"ID":"ea69f37b-b923-4f5b-b8a3-e7e91d20165b","Type":"ContainerStarted","Data":"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a"} Oct 30 22:24:13.550822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.545340 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" event={"ID":"ea69f37b-b923-4f5b-b8a3-e7e91d20165b","Type":"ContainerStarted","Data":"a1f32bcc607352c90e509937b4bfe553d9aaaf16d62d5ef7b923b4914a54b2d7"} Oct 30 22:24:13.550822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.546475 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:13.678456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.678343 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" podStartSLOduration=2.808879537 podStartE2EDuration="3.678322328s" totalImagesPullingTime="869.442791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:13.661248381 +0000 UTC m=+1374.256336688" watchObservedRunningTime="2026-10-30 22:24:13.678322328 +0000 UTC m=+1374.273410635" Oct 30 22:24:13.937873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.927974 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:13.937873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:13.928035 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:14.457823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:14.450147 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:24:14.457823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:14.450205 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:24:14.922261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:14.922183 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:24:14.922261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:14.922236 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:24:16.060367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.060329 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:16.062083 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.062057 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:16.063744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.063721 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:24:16.122036 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:24:16.122002 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:24:16.188016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.187985 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:16.188689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.188661 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:16.352870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:16.343876 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:24:20.413981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.413937 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:24:20.419983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.419955 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.469006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.468981 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dee83803-serving-cert\"" Oct 30 22:24:20.505904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.505868 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.506083 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.505926 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4qk\" (UniqueName: \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.506343 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.506315 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.506477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.506452 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.519419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.519393 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:24:20.580166 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.580140 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:24:20.607556 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.607519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.607721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.607578 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.607721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.607615 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4qk\" (UniqueName: \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.607938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.607915 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.608077 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:24:20.608058 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-serving-cert: secret "maas-api-e2e-ait-dee83803-serving-cert" not found Oct 30 22:24:20.608155 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:24:20.608141 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls podName:e3abc4a8-66c4-42ea-9dee-323b108fe865 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:21.108111901 +0000 UTC m=+1381.703200188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls") pod "maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" (UID: "e3abc4a8-66c4-42ea-9dee-323b108fe865") : secret "maas-api-e2e-ait-dee83803-serving-cert" not found Oct 30 22:24:20.608424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.608405 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.610219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.610201 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:20.782514 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:20.782475 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4qk\" (UniqueName: \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:21.122490 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:21.122404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:21.124852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:21.124807 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") pod \"maas-api-e2e-ait-dee83803-64f8bbd449-dp78g\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:21.336745 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:21.336709 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:21.717720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:21.716888 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:24:21.723628 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:24:21.723590 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3abc4a8_66c4_42ea_9dee_323b108fe865.slice/crio-55fe05cc9efbc0923b8b6b0f4d0d0c0680e13f3dee397f36690024e0fbffa968 WatchSource:0}: Error finding container 55fe05cc9efbc0923b8b6b0f4d0d0c0680e13f3dee397f36690024e0fbffa968: Status 404 returned error can't find the container with id 55fe05cc9efbc0923b8b6b0f4d0d0c0680e13f3dee397f36690024e0fbffa968 Oct 30 22:24:22.638479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:22.638443 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" event={"ID":"e3abc4a8-66c4-42ea-9dee-323b108fe865","Type":"ContainerStarted","Data":"55fe05cc9efbc0923b8b6b0f4d0d0c0680e13f3dee397f36690024e0fbffa968"} Oct 30 22:24:23.647202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:23.647162 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" event={"ID":"e3abc4a8-66c4-42ea-9dee-323b108fe865","Type":"ContainerStarted","Data":"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e"} Oct 30 22:24:23.647612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:23.647246 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:23.761294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:23.761228 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" podStartSLOduration=2.858632858 podStartE2EDuration="3.76121363s" totalImagesPullingTime="902.580772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:23.757535351 +0000 UTC m=+1384.352623676" watchObservedRunningTime="2026-10-30 22:24:23.76121363 +0000 UTC m=+1384.356301935" Oct 30 22:24:29.658949 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:29.658918 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:24:46.405212 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:24:46.405005 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:24:56.781675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.781622 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.852221 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.887923 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.887999 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888040 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b69q\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888149 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888190 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888225 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888299 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.893823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.888348 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:56.940562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:56.940519 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.005888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.005961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006006 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006093 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006150 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006392 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006438 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b69q\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.006499 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.007022 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.008382 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.026946 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.033881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.027304 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.074694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.062557 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-05e3625f-openshift-default-dockercfg-8l4lr\"" Oct 30 22:24:57.074694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.063284 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.074694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.063607 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.074694 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.069740 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.129189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.129154 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.129369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.129202 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b69q\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.241390 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.241353 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:57.676313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.676212 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:24:57.680289 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:24:57.680260 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9a8d6c_21eb_4cb0_a2e1_ba93bdcc179c.slice/crio-4ba0fcd7ed6e93f264b910ca5abc60c1897bf46062fea384d1a83ad4ea4a5de7 WatchSource:0}: Error finding container 4ba0fcd7ed6e93f264b910ca5abc60c1897bf46062fea384d1a83ad4ea4a5de7: Status 404 returned error can't find the container with id 4ba0fcd7ed6e93f264b910ca5abc60c1897bf46062fea384d1a83ad4ea4a5de7 Oct 30 22:24:57.683068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.683031 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.683157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.683126 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.683200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.683158 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.966091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.966058 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerID="3c118426b08b5398213f88ba52eeb708521fb6e0b0e97a22756f74775daae4dd" exitCode=0 Oct 30 22:24:57.966539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.966130 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerDied","Data":"3c118426b08b5398213f88ba52eeb708521fb6e0b0e97a22756f74775daae4dd"} Oct 30 22:24:57.966667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.966652 4305 scope.go:122] "RemoveContainer" containerID="3c118426b08b5398213f88ba52eeb708521fb6e0b0e97a22756f74775daae4dd" Oct 30 22:24:57.968230 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.968157 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" event={"ID":"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c","Type":"ContainerStarted","Data":"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1"} Oct 30 22:24:57.968230 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.968205 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" event={"ID":"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c","Type":"ContainerStarted","Data":"4ba0fcd7ed6e93f264b910ca5abc60c1897bf46062fea384d1a83ad4ea4a5de7"} Oct 30 22:24:57.968490 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.968462 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.968552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.968537 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.968587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:57.968577 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:58.215174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:58.205663 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podStartSLOduration=2.205642889 podStartE2EDuration="2.205642889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:58.195953155 +0000 UTC m=+1418.791041487" watchObservedRunningTime="2026-10-30 22:24:58.205642889 +0000 UTC m=+1418.800731195" Oct 30 22:24:58.242344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:58.242254 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:24:58.243736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:58.243714 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:24:58.243858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:58.243759 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:24:58.997821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:58.997497 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerStarted","Data":"c12dd4afb9cc20174f8e33ba64c1b1f854c1c13ad39990233d497c871037678a"} Oct 30 22:24:59.242333 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:59.242297 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:24:59.242620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:59.242586 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:24:59.742030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:59.742000 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:24:59.742277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:59.742106 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:24:59.742393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:24:59.742376 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:25:00.242608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:00.242575 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.243055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:00.242628 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:25:00.742040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:00.742009 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.742206 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:00.742057 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:25:01.241867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:01.241828 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:25:01.242034 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:01.241886 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:25:01.741530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:01.741493 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:25:01.742058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:01.741553 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:25:02.242392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:02.242358 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:25:02.242565 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:02.242417 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:25:02.741362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:02.741320 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:25:02.741537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:02.741377 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:25:03.254821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:03.249311 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:25:03.254821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:03.249384 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:25:03.741859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:03.741828 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:25:03.742027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:03.741880 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:25:04.246303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.246268 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:25:04.247017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.246989 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:25:04.247643 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.247619 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:25:04.746072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.746042 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:25:04.746557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.746538 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:25:04.747150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:04.747127 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:25:16.293820 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:16.293076 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:25:41.850073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.850034 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:41.854661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.854639 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.956963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.956924 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:41.973276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973238 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.973466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973391 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.973466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973441 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.973573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973560 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.973627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973612 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.973757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.973741 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.974054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.974032 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.974136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.974076 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77br\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:41.974182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:41.974154 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.075864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.075746 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.075864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.075846 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77br\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.075889 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.075933 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076025 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076052 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076117 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076172 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076252 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.076645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076596 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.077002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076678 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.077002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076712 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.077002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076920 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.077002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.076976 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.078639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.078616 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.078639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.078633 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.134131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.134067 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.134308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.134284 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77br\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.167658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.167637 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:42.738152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.738119 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:42.742070 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:25:42.742039 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac479e2_6ced_417f_998c_0ea245fe773a.slice/crio-c523b8d5769854ccba30ead46ac7a85e9606a1bbcb7729df158339d4698d3025 WatchSource:0}: Error finding container c523b8d5769854ccba30ead46ac7a85e9606a1bbcb7729df158339d4698d3025: Status 404 returned error can't find the container with id c523b8d5769854ccba30ead46ac7a85e9606a1bbcb7729df158339d4698d3025 Oct 30 22:25:42.744622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.744588 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:42.744707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.744655 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:42.744707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:42.744682 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:43.508393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:43.508350 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" event={"ID":"fac479e2-6ced-417f-998c-0ea245fe773a","Type":"ContainerStarted","Data":"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950"} Oct 30 22:25:43.508903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:43.508417 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" event={"ID":"fac479e2-6ced-417f-998c-0ea245fe773a","Type":"ContainerStarted","Data":"c523b8d5769854ccba30ead46ac7a85e9606a1bbcb7729df158339d4698d3025"} Oct 30 22:25:43.673947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:43.673896 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podStartSLOduration=2.673881128 podStartE2EDuration="2.673881128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:43.67148296 +0000 UTC m=+1464.266571288" watchObservedRunningTime="2026-10-30 22:25:43.673881128 +0000 UTC m=+1464.268969495" Oct 30 22:25:44.171038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:44.170955 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:44.172268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:44.172241 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:25:44.172434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:44.172287 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:25:45.170837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:45.169788 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:25:45.170837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:45.169893 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:25:45.997256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:45.997221 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:25:45.997473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:45.997442 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" podUID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" containerName="maas-api" containerID="cri-o://1f9e63ec3f8bb6c43afe43f6286bcd84e7c26a4dfdfa721c7a6edd4a1179e099" gracePeriod=30 Oct 30 22:25:46.139857 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:46.136559 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:25:46.177762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:46.177613 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:25:46.177762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:46.177669 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:25:46.822160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:46.822109 4305 generic.go:356] "Generic (PLEG): container finished" podID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" containerID="1f9e63ec3f8bb6c43afe43f6286bcd84e7c26a4dfdfa721c7a6edd4a1179e099" exitCode=0 Oct 30 22:25:46.822244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:46.822187 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" event={"ID":"c67fafa8-7134-4fa4-bab4-51ccf9672d94","Type":"ContainerDied","Data":"1f9e63ec3f8bb6c43afe43f6286bcd84e7c26a4dfdfa721c7a6edd4a1179e099"} Oct 30 22:25:46.848002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:46.847975 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:25:47.091542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.087443 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\") pod \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " Oct 30 22:25:47.091542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.087552 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\") pod \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " Oct 30 22:25:47.091542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.087663 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\" (UniqueName: \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\") pod \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " Oct 30 22:25:47.091542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.087928 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\") pod \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\" (UID: \"c67fafa8-7134-4fa4-bab4-51ccf9672d94\") " Oct 30 22:25:47.121459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.120704 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca" pod "c67fafa8-7134-4fa4-bab4-51ccf9672d94" (UID: "c67fafa8-7134-4fa4-bab4-51ccf9672d94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:47.196828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.183940 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:25:47.196828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.184015 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:25:47.224820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.211649 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67fafa8-7134-4fa4-bab4-51ccf9672d94-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.260831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.258642 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc" pod "c67fafa8-7134-4fa4-bab4-51ccf9672d94" (UID: "c67fafa8-7134-4fa4-bab4-51ccf9672d94"). InnerVolumeSpecName "kube-api-access-7gqxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:47.260831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.259902 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls" pod "c67fafa8-7134-4fa4-bab4-51ccf9672d94" (UID: "c67fafa8-7134-4fa4-bab4-51ccf9672d94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:47.260831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.259999 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs" pod "c67fafa8-7134-4fa4-bab4-51ccf9672d94" (UID: "c67fafa8-7134-4fa4-bab4-51ccf9672d94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:47.313773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.313737 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.313773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.313772 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqxc\" (UniqueName: \"kubernetes.io/projected/c67fafa8-7134-4fa4-bab4-51ccf9672d94-kube-api-access-7gqxc\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.314002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.313788 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67fafa8-7134-4fa4-bab4-51ccf9672d94-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:47.829998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.829915 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" Oct 30 22:25:47.829998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.829948 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp" event={"ID":"c67fafa8-7134-4fa4-bab4-51ccf9672d94","Type":"ContainerDied","Data":"75799a9f6dd5111e8826165f100104faecb5c125b4b89eb115772b77702233cc"} Oct 30 22:25:47.830237 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:47.830024 4305 scope.go:122] "RemoveContainer" containerID="1f9e63ec3f8bb6c43afe43f6286bcd84e7c26a4dfdfa721c7a6edd4a1179e099" Oct 30 22:25:48.011824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.006740 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:25:48.070868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.069913 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-05e3625f-7cfdc7df69-mflpp"] Oct 30 22:25:48.171732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.171176 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:25:48.171732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.171235 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:25:48.355906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.355870 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:25:48.356663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.356628 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" containerID="cri-o://b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1" gracePeriod=30 Oct 30 22:25:48.382607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.382572 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:48.398358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:48.398289 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" containerID="cri-o://7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950" gracePeriod=30 Oct 30 22:25:49.249167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:49.249123 4305 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:49.249330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:49.249209 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:49.520531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:49.520451 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" path="/var/lib/kubelet/pods/c67fafa8-7134-4fa4-bab4-51ccf9672d94/volumes" Oct 30 22:25:52.368450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.368416 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:25:52.368959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.368683 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" podUID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" containerName="maas-api" containerID="cri-o://e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a" gracePeriod=30 Oct 30 22:25:52.960569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.960184 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:25:53.006695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.999523 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\") pod \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " Oct 30 22:25:53.006695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.999659 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\" (UniqueName: \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\") pod \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " Oct 30 22:25:53.006695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.999761 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\") pod \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " Oct 30 22:25:53.006695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:52.999810 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\") pod \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\" (UID: \"ea69f37b-b923-4f5b-b8a3-e7e91d20165b\") " Oct 30 22:25:53.006695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.001955 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca" pod "ea69f37b-b923-4f5b-b8a3-e7e91d20165b" (UID: "ea69f37b-b923-4f5b-b8a3-e7e91d20165b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:53.060862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.059979 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls" pod "ea69f37b-b923-4f5b-b8a3-e7e91d20165b" (UID: "ea69f37b-b923-4f5b-b8a3-e7e91d20165b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:53.095890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.084993 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs" pod "ea69f37b-b923-4f5b-b8a3-e7e91d20165b" (UID: "ea69f37b-b923-4f5b-b8a3-e7e91d20165b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:53.095890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.085779 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz" pod "ea69f37b-b923-4f5b-b8a3-e7e91d20165b" (UID: "ea69f37b-b923-4f5b-b8a3-e7e91d20165b"). InnerVolumeSpecName "kube-api-access-wwvqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107193 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvqz\" (UniqueName: \"kubernetes.io/projected/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-kube-api-access-wwvqz\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107232 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107252 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107267 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea69f37b-b923-4f5b-b8a3-e7e91d20165b-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107542 4305 generic.go:356] "Generic (PLEG): container finished" podID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" containerID="e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a" exitCode=0 Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107615 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" event={"ID":"ea69f37b-b923-4f5b-b8a3-e7e91d20165b","Type":"ContainerDied","Data":"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a"} Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107665 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" event={"ID":"ea69f37b-b923-4f5b-b8a3-e7e91d20165b","Type":"ContainerDied","Data":"a1f32bcc607352c90e509937b4bfe553d9aaaf16d62d5ef7b923b4914a54b2d7"} Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107699 4305 scope.go:122] "RemoveContainer" containerID="e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a" Oct 30 22:25:53.109818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.107921 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6" Oct 30 22:25:53.219547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.219520 4305 scope.go:122] "RemoveContainer" containerID="e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a" Oct 30 22:25:53.219963 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:53.219937 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a\": container with ID starting with e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a not found: ID does not exist" containerID="e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a" Oct 30 22:25:53.220096 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.219976 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a"} err="failed to get container status \"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a\": rpc error: code = NotFound desc = could not find container \"e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a\": container with ID starting with e10b095233237c6ff751c371f9dc39cb877c59487fa76b018ff3b33ae26b4d0a not found: ID does not exist" Oct 30 22:25:53.376874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.374907 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:25:53.392829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.392183 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae20a62d-86bb56594d-l4zf6"] Oct 30 22:25:53.573117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.573080 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" path="/var/lib/kubelet/pods/ea69f37b-b923-4f5b-b8a3-e7e91d20165b/volumes" Oct 30 22:25:53.956832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:53.956781 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:25:54.042887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.042852 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.042939 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.042997 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043025 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043116 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043235 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043283 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043307 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.043344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043323 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.043720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043352 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:54.043720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043463 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.043720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043470 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.043720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.043625 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.044125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.044107 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.044185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.044131 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.044185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.044144 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.044185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.044154 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.044185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.044163 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.045758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.045734 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:54.045963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.045940 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "kube-api-access-5b69q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.046031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.045976 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.116613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.116529 4305 generic.go:356] "Generic (PLEG): container finished" podID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerID="b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1" exitCode=0 Oct 30 22:25:54.116613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.116581 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" Oct 30 22:25:54.116613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.116583 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" event={"ID":"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c","Type":"ContainerDied","Data":"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1"} Oct 30 22:25:54.116913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.116661 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr" event={"ID":"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c","Type":"ContainerDied","Data":"4ba0fcd7ed6e93f264b910ca5abc60c1897bf46062fea384d1a83ad4ea4a5de7"} Oct 30 22:25:54.116913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.116695 4305 scope.go:122] "RemoveContainer" containerID="b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1" Oct 30 22:25:54.127683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.127657 4305 scope.go:122] "RemoveContainer" containerID="b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1" Oct 30 22:25:54.127986 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:54.127967 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1\": container with ID starting with b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1 not found: ID does not exist" containerID="b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1" Oct 30 22:25:54.128108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.127995 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1"} err="failed to get container status \"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1\": rpc error: code = NotFound desc = could not find container \"b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1\": container with ID starting with b6b19cf2f3d2e361dbd9cfee99ccf4f78af97cf8dbfd4a6500d5dc5cfc9225d1 not found: ID does not exist" Oct 30 22:25:54.144941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.144920 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\") pod \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\" (UID: \"2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c\") " Oct 30 22:25:54.145383 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.145362 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b69q\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-kube-api-access-5b69q\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.145383 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.145381 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.145512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.145390 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.147006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.146984 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy" pod "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" (UID: "2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.246989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.246960 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.535488 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.535321 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:25:54.556596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.556565 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-js5jr"] Oct 30 22:25:54.782754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.782724 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:54.872722 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.872683 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.872950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.872750 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.872950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.872866 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.872959 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873024 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873070 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873105 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873144 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873187 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873213 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\") pod \"fac479e2-6ced-417f-998c-0ea245fe773a\" (UID: \"fac479e2-6ced-417f-998c-0ea245fe773a\") " Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873277 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873337 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873429 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.873882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.873605 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:54.874362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.874041 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.874362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.874067 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.874362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.874083 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.874362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.874098 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.874362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.874113 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac479e2-6ced-417f-998c-0ea245fe773a-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.875354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.875327 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:54.875354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.875341 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "kube-api-access-l77br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.875649 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.875625 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.875741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.875651 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token" pod "fac479e2-6ced-417f-998c-0ea245fe773a" (UID: "fac479e2-6ced-417f-998c-0ea245fe773a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.975829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.975735 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac479e2-6ced-417f-998c-0ea245fe773a-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.975829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.975769 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77br\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-kube-api-access-l77br\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.975829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.975781 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac479e2-6ced-417f-998c-0ea245fe773a-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.975829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:54.975793 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac479e2-6ced-417f-998c-0ea245fe773a-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:55.125466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.125427 4305 generic.go:356] "Generic (PLEG): container finished" podID="fac479e2-6ced-417f-998c-0ea245fe773a" containerID="7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950" exitCode=0 Oct 30 22:25:55.125755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.125479 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" event={"ID":"fac479e2-6ced-417f-998c-0ea245fe773a","Type":"ContainerDied","Data":"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950"} Oct 30 22:25:55.125755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.125525 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" event={"ID":"fac479e2-6ced-417f-998c-0ea245fe773a","Type":"ContainerDied","Data":"c523b8d5769854ccba30ead46ac7a85e9606a1bbcb7729df158339d4698d3025"} Oct 30 22:25:55.125755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.125547 4305 scope.go:122] "RemoveContainer" containerID="7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950" Oct 30 22:25:55.125755 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.125484 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw" Oct 30 22:25:55.136529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.136509 4305 scope.go:122] "RemoveContainer" containerID="7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950" Oct 30 22:25:55.136839 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:55.136820 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950\": container with ID starting with 7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950 not found: ID does not exist" containerID="7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950" Oct 30 22:25:55.136919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.136851 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950"} err="failed to get container status \"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950\": rpc error: code = NotFound desc = could not find container \"7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950\": container with ID starting with 7ccd40e2ede9a2a6484058a2491527a96f1a70c85171545db3a235673d082950 not found: ID does not exist" Oct 30 22:25:55.284055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.283969 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:55.328840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.328807 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-dzmqw"] Oct 30 22:25:55.520387 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.520358 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" path="/var/lib/kubelet/pods/2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c/volumes" Oct 30 22:25:55.520871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:55.520857 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" path="/var/lib/kubelet/pods/fac479e2-6ced-417f-998c-0ea245fe773a/volumes" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.562928 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563867 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563889 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563911 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563918 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563935 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" containerName="maas-api" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563944 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" containerName="maas-api" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563968 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" containerName="maas-api" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.563975 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" containerName="maas-api" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.564152 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9a8d6c-21eb-4cb0-a2e1-ba93bdcc179c" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.564163 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac479e2-6ced-417f-998c-0ea245fe773a" containerName="istio-proxy" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.564173 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea69f37b-b923-4f5b-b8a3-e7e91d20165b" containerName="maas-api" Oct 30 22:25:58.567929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.564185 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67fafa8-7134-4fa4-bab4-51ccf9672d94" containerName="maas-api" Oct 30 22:25:58.669917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.669874 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.706906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.704941 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.763616 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.763677 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.763774 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764084 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764210 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764282 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcw8\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764325 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764371 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.764402 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.813212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.784450 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e8ed1149-openshift-default-dockercfg-7bv4n\"" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.921961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcw8\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922227 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922266 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922294 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922370 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.922482 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.929875 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.930157 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.939821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.931015 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.941581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.941533 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.942158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.942135 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:58.994218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:58.961564 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:25:59.075076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.075027 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcw8\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:59.077188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.076968 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:59.082352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.082320 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:59.109548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.109427 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\") pod \"e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:59.121746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.120781 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.369277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.369194 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:25:59.369789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.369749 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerName="maas-api" containerID="cri-o://fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e" gracePeriod=30 Oct 30 22:25:59.419945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.419904 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:25:59.459080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.459051 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.460887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.460861 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:25:59.462446 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462417 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.462559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462479 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.462677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462654 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxnf\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462750 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462861 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.462977 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.463080 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.463141 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.463255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.463164 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.464247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.464228 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:25:59.471269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.471235 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:25:59.566058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.566018 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxnf\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.594187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.594151 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.594850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.594821 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.595109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.595089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.602039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.602013 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.602249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.602232 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.602370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.602356 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.602661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.602646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.603990 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.603956 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.604318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.595161 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.604428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.603126 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.604784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.604763 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.608664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.594897 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.609602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.608322 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.609749 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.608494 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.646717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.644548 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.665575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.665529 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.79:8443/health\": dial tcp 10.134.0.79:8443: connect: connection refused" Oct 30 22:25:59.687574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.687533 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxnf\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.688063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.688036 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-v7krd\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.688503 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:25:59.688473 4305 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3abc4a8_66c4_42ea_9dee_323b108fe865.slice/crio-conmon-fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:25:59.705891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.705404 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.705891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.705555 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.705891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.705682 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.705891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.705762 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.712328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.708089 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.712328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.708172 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr98\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.712328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.708265 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.712328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.708311 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.712328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.708340 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810135 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810343 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr98\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810468 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810551 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810613 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810726 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.810900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.810888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.811157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.811137 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.811346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.811325 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.811536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.811312 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.811656 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.811564 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.811894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.811870 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.812160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.812138 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.814221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.814197 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.814317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.814299 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.815350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.815325 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.872015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.871985 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.872521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.872503 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr98\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bdh5v\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:25:59.879167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.877711 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:25:59.893389 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:25:59.893368 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:26:00.059823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.058718 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:26:00.176085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.172473 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.176085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.172578 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.176085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.172621 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.288503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.288179 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:26:00.343700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.343664 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") pod \"e3abc4a8-66c4-42ea-9dee-323b108fe865\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " Oct 30 22:26:00.343994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.343973 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\" (UniqueName: \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\") pod \"e3abc4a8-66c4-42ea-9dee-323b108fe865\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " Oct 30 22:26:00.344157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.344136 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\") pod \"e3abc4a8-66c4-42ea-9dee-323b108fe865\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " Oct 30 22:26:00.344234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.344219 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\") pod \"e3abc4a8-66c4-42ea-9dee-323b108fe865\" (UID: \"e3abc4a8-66c4-42ea-9dee-323b108fe865\") " Oct 30 22:26:00.345967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.345930 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca" pod "e3abc4a8-66c4-42ea-9dee-323b108fe865" (UID: "e3abc4a8-66c4-42ea-9dee-323b108fe865"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:00.348219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.348174 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk" pod "e3abc4a8-66c4-42ea-9dee-323b108fe865" (UID: "e3abc4a8-66c4-42ea-9dee-323b108fe865"). InnerVolumeSpecName "kube-api-access-6f4qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:00.349121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.349098 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs" pod "e3abc4a8-66c4-42ea-9dee-323b108fe865" (UID: "e3abc4a8-66c4-42ea-9dee-323b108fe865"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:00.349305 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.349277 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls" pod "e3abc4a8-66c4-42ea-9dee-323b108fe865" (UID: "e3abc4a8-66c4-42ea-9dee-323b108fe865"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:00.446579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.446545 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3abc4a8-66c4-42ea-9dee-323b108fe865-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.446579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.446578 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.446769 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.446588 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3abc4a8-66c4-42ea-9dee-323b108fe865-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.446769 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.446596 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4qk\" (UniqueName: \"kubernetes.io/projected/e3abc4a8-66c4-42ea-9dee-323b108fe865-kube-api-access-6f4qk\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.464455 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.464421 4305 generic.go:356] "Generic (PLEG): container finished" podID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerID="fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e" exitCode=0 Oct 30 22:26:00.464695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.464460 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" event={"ID":"e3abc4a8-66c4-42ea-9dee-323b108fe865","Type":"ContainerDied","Data":"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e"} Oct 30 22:26:00.464877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.464855 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" event={"ID":"e3abc4a8-66c4-42ea-9dee-323b108fe865","Type":"ContainerDied","Data":"55fe05cc9efbc0923b8b6b0f4d0d0c0680e13f3dee397f36690024e0fbffa968"} Oct 30 22:26:00.464970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.464493 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g" Oct 30 22:26:00.464970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.464897 4305 scope.go:122] "RemoveContainer" containerID="fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e" Oct 30 22:26:00.467916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.467848 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerStarted","Data":"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac"} Oct 30 22:26:00.467916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.467900 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerStarted","Data":"d169984cd9d5acc1428503e16cdbfc4daf6bac623e788b88bb7d249e8fbbd53e"} Oct 30 22:26:00.477373 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.477357 4305 scope.go:122] "RemoveContainer" containerID="fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e" Oct 30 22:26:00.477599 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:00.477580 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e\": container with ID starting with fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e not found: ID does not exist" containerID="fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e" Oct 30 22:26:00.477668 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.477613 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e"} err="failed to get container status \"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e\": rpc error: code = NotFound desc = could not find container \"fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e\": container with ID starting with fb20e94679b97f05c1e2247afb91a4d27962d24c33aca94e406a6378ce2f455e not found: ID does not exist" Oct 30 22:26:00.625937 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.625904 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:26:00.636415 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.636374 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dee83803-64f8bbd449-dp78g"] Oct 30 22:26:00.643541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.643302 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:26:00.644614 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:00.644585 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da63901_13b2_489c_82de_42e638560747.slice/crio-29053804aa8e2a974e9e384bc9013f9f5af21c80d6a339a4e94d5da68ef297d5 WatchSource:0}: Error finding container 29053804aa8e2a974e9e384bc9013f9f5af21c80d6a339a4e94d5da68ef297d5: Status 404 returned error can't find the container with id 29053804aa8e2a974e9e384bc9013f9f5af21c80d6a339a4e94d5da68ef297d5 Oct 30 22:26:00.647479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.647422 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.647568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.647532 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.647616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.647573 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.662662 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.662622 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:26:00.667038 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:00.667009 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0829a02d_a461_46cc_9452_8216602a604b.slice/crio-cbb919c8b091516606c44ab5d5ace00534dd66d2b766046f08b4f4ca571d6902 WatchSource:0}: Error finding container cbb919c8b091516606c44ab5d5ace00534dd66d2b766046f08b4f4ca571d6902: Status 404 returned error can't find the container with id cbb919c8b091516606c44ab5d5ace00534dd66d2b766046f08b4f4ca571d6902 Oct 30 22:26:00.670549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.670517 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.670657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.670596 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.670657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.670642 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.743789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:00.743739 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podStartSLOduration=2.743723176 podStartE2EDuration="2.743723176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:00.739479544 +0000 UTC m=+1481.334567861" watchObservedRunningTime="2026-10-30 22:26:00.743723176 +0000 UTC m=+1481.338811483" Oct 30 22:26:01.126825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.123097 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:26:01.148822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.138734 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerName="maas-api" Oct 30 22:26:01.148822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.138766 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerName="maas-api" Oct 30 22:26:01.148822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.138997 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" containerName="maas-api" Oct 30 22:26:01.398871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.379021 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:26:01.398871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.379155 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.572304 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.474418 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-926fc6-openshift-default-dockercfg-4cdc6\"" Oct 30 22:26:01.572304 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.505462 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:26:01.586398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.586365 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3abc4a8-66c4-42ea-9dee-323b108fe865" path="/var/lib/kubelet/pods/e3abc4a8-66c4-42ea-9dee-323b108fe865/volumes" Oct 30 22:26:01.600527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.600488 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.601478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.601449 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.601705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.601686 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2cd\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.601895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.601877 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.602115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.602098 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.602251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.602235 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.602448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.602431 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.602647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.602631 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.602746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.602734 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.628008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.627894 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:26:01.662007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.661616 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" event={"ID":"0829a02d-a461-46cc-9452-8216602a604b","Type":"ContainerStarted","Data":"26315c6cdf9788f1e14cdf3fd72fc06184e03608d44ce9197ed3332172c9b6c7"} Oct 30 22:26:01.662500 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.662468 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" event={"ID":"0829a02d-a461-46cc-9452-8216602a604b","Type":"ContainerStarted","Data":"cbb919c8b091516606c44ab5d5ace00534dd66d2b766046f08b4f4ca571d6902"} Oct 30 22:26:01.707606 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.707567 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.710680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.710648 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.711055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.711032 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.711210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.711193 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.717332 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.717295 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.720903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.720877 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.721122 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.721104 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2cd\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.721308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.721295 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.721555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.721528 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.721988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.714476 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.724231 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.714853 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.724383 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.714001 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.724760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.717778 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.726379 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.725515 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:01.731862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.731822 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:26:01.732236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.732211 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" containerID="cri-o://c12dd4afb9cc20174f8e33ba64c1b1f854c1c13ad39990233d497c871037678a" gracePeriod=30 Oct 30 22:26:01.767548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.763778 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" event={"ID":"8da63901-13b2-489c-82de-42e638560747","Type":"ContainerStarted","Data":"c40bf2f1ac9c5a3582cbba4e2df27bfe9f6917586176373e9f7e597088c6a4d6"} Oct 30 22:26:01.767548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.763897 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" event={"ID":"8da63901-13b2-489c-82de-42e638560747","Type":"ContainerStarted","Data":"29053804aa8e2a974e9e384bc9013f9f5af21c80d6a339a4e94d5da68ef297d5"} Oct 30 22:26:01.767548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.764498 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:26:01.777856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.773439 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:26:01.895930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.890857 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:26:01.920132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.912027 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:26:01.929584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.929542 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:01.929886 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.929839 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:01.930080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.929888 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:01.930224 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:01.930205 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:02.192871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.185253 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:02.263841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.249988 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:02.265195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.265130 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:02.289031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.288782 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2cd\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\") pod \"e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:02.356498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.356432 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podStartSLOduration=4.356411736 podStartE2EDuration="4.356411736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:02.354393773 +0000 UTC m=+1482.949482080" watchObservedRunningTime="2026-10-30 22:26:02.356411736 +0000 UTC m=+1482.951500043" Oct 30 22:26:02.367577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.367506 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podStartSLOduration=4.3674868270000005 podStartE2EDuration="4.367486827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:02.224165196 +0000 UTC m=+1482.819253503" watchObservedRunningTime="2026-10-30 22:26:02.367486827 +0000 UTC m=+1482.962575134" Oct 30 22:26:02.381653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.381147 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:02.906661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.906630 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:02.907624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.906684 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:02.907624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.907177 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:02.907624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.907215 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:02.911640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.910149 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:26:02.911640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.910929 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:02.911640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.911005 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:02.911640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:02.911044 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:03.813431 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.811968 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerStarted","Data":"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba"} Oct 30 22:26:03.813431 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.812054 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerStarted","Data":"6cbec83833fb2f7e275ecd0798a290595cb76f10b3d9601ce127f43713c6cd70"} Oct 30 22:26:03.893838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.892959 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:03.893838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.893019 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:03.899247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.899118 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:03.899247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.899171 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:03.933371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:03.933312 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podStartSLOduration=2.933298119 podStartE2EDuration="2.933298119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:03.930578193 +0000 UTC m=+1484.525666498" watchObservedRunningTime="2026-10-30 22:26:03.933298119 +0000 UTC m=+1484.528386498" Oct 30 22:26:04.386853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.386256 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:04.396229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.396190 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:04.763821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.759286 4305 patch_prober.go:29] interesting pod/e2e-ait-dee83803-openshift-default-54775796bf-8vttp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:04.763821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.759347 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:04.873820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.870758 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:04.889825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.878864 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:04.889825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.878923 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:04.889825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.882039 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:26:04.919241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.919200 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:04.919487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:04.919459 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:05.897822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:05.891006 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:05.897822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:05.891078 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:05.899056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:05.899026 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:05.899281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:05.899254 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:06.898861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:06.894207 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:06.898861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:06.894288 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:06.917938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:06.917883 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:06.918292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:06.918193 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:07.307821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.306034 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:07.411871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.410339 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:07.429823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.423057 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:07.523823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.523728 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 22:26:07.539825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.527974 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 22:26:07.700033 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.680880 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lhj8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:07.700033 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.680981 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:07.784222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.776286 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:07.903865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.897660 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:07.903865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.897735 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:07.903865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.901606 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:07.903865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:07.901654 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:08.250258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.250217 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:26:08.319881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.319837 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:08.320439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.320415 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:26:08.336882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.336852 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerID="c12dd4afb9cc20174f8e33ba64c1b1f854c1c13ad39990233d497c871037678a" exitCode=0 Oct 30 22:26:08.337085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.337070 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerDied","Data":"c12dd4afb9cc20174f8e33ba64c1b1f854c1c13ad39990233d497c871037678a"} Oct 30 22:26:08.337205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.337193 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" event={"ID":"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d","Type":"ContainerDied","Data":"d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965"} Oct 30 22:26:08.337290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.337279 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ff0c513db827a03c07fceb87285ce4841f30313eb3d884b6a329f42bb40965" Oct 30 22:26:08.337375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.337366 4305 scope.go:122] "RemoveContainer" containerID="3c118426b08b5398213f88ba52eeb708521fb6e0b0e97a22756f74775daae4dd" Oct 30 22:26:08.339055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.339036 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:26:08.430111 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.430086 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:26:08.434627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434594 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.434756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434691 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.434756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434738 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.434868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434851 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.434902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434885 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.434935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.434920 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.435103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435082 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.435208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435193 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.435268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435226 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\") pod \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\" (UID: \"ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d\") " Oct 30 22:26:08.435919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435894 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.436041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435920 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.436149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.435964 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:08.436235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.436122 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.436328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.436171 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.436615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.436595 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.442547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.442217 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.442821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.442785 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "kube-api-access-8ddf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:08.444021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.443996 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:08.466269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.461995 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token" pod "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" (UID: "ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:08.538013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.537984 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538231 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538329 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538436 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538525 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538515 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538585 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ddf2\" (UniqueName: \"kubernetes.io/projected/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-kube-api-access-8ddf2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538655 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.538741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.538729 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.896822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.888831 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:08.896822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.888894 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:08.897139 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.897103 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:08.897205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:08.897169 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:09.365131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.361103 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp" Oct 30 22:26:09.594295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.594255 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:26:09.600966 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.600935 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dee83803-openshift-default-54775796bf-8vttp"] Oct 30 22:26:09.878633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.878595 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:09.878810 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.878662 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:09.894105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.894070 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:09.894272 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:09.894122 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:10.878280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:10.878236 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:26:10.878816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:10.878303 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:26:10.894268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:10.894233 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:26:10.894399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:10.894302 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:26:12.015842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:11.979577 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" path="/var/lib/kubelet/pods/ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d/volumes" Oct 30 22:26:12.897016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:12.896979 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:12.897231 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:12.897047 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:12.936332 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:12.936281 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:12.936495 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:12.936374 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:13.919864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:13.919825 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:13.920528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:13.920479 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:13.970563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:13.970524 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:13.971043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:13.970997 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:14.030069 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.030025 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:26:14.031171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031147 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.031227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031177 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.031227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031196 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.031227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031205 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.031406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031391 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.031438 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.031411 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab096bfd-3e6e-4abf-bfdb-0bdb90334a3d" containerName="istio-proxy" Oct 30 22:26:14.035955 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.035932 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.123988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.123901 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:26:14.168342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168292 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168518 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168374 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168518 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168426 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxn6\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168518 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168483 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168552 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168605 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168746 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.168946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168920 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.169028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.168994 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.270915 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271081 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271138 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271194 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271247 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271289 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxn6\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271346 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271411 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271462 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.271822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271536 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.272522 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.271930 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.272522 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.272421 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.296821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.272900 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.296821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.273659 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.296821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.276320 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.296821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.290260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.330126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.330092 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.330265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.330228 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxn6\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n42cb\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.352366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.352283 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:14.823199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.823165 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:26:14.827461 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:14.827437 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0767789_10bd_4198_a644_3a7b441e4f03.slice/crio-0d396f57a8d5fd2b312a754973491fd95ab163c42a75d2d964561ace1c9ef2fc WatchSource:0}: Error finding container 0d396f57a8d5fd2b312a754973491fd95ab163c42a75d2d964561ace1c9ef2fc: Status 404 returned error can't find the container with id 0d396f57a8d5fd2b312a754973491fd95ab163c42a75d2d964561ace1c9ef2fc Oct 30 22:26:14.830043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.830011 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:14.830117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.830077 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:14.830117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.830107 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:14.889510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.889475 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:26:14.898506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.897567 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:26:14.966249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.965885 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:14.966249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:14.965949 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:15.019037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:15.018875 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:15.019037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:15.018951 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:15.908912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:15.905082 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" event={"ID":"f0767789-10bd-4198-a644-3a7b441e4f03","Type":"ContainerStarted","Data":"75656d8358dd70356a9aadc175ef5265a130d3637b3ff3c8d916c2c9f758f80d"} Oct 30 22:26:15.908912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:15.905158 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" event={"ID":"f0767789-10bd-4198-a644-3a7b441e4f03","Type":"ContainerStarted","Data":"0d396f57a8d5fd2b312a754973491fd95ab163c42a75d2d964561ace1c9ef2fc"} Oct 30 22:26:16.028825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.027928 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-v7krd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:16.028825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.028009 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:16.066987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.066936 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bdh5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:16.067273 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.067229 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:16.260821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.243697 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:26:16.260821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.251303 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:26:16.265647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.265556 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podStartSLOduration=3.265536854 podStartE2EDuration="3.265536854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:16.086552977 +0000 UTC m=+1496.681641325" watchObservedRunningTime="2026-10-30 22:26:16.265536854 +0000 UTC m=+1496.860625163" Oct 30 22:26:16.283883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.283854 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:26:16.311407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.311377 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:26:16.311828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.311714 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.323347 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:16.318382 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:26:16.368842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.358185 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:26:16.370212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.369913 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-7c5d18-openshift-default-dockercfg-46hhb\"" Oct 30 22:26:16.387218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.382910 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:16.387218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.383064 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.387218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.386923 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:16.395201 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.395170 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.455989 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nbc\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456121 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456209 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456247 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456328 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456463 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456512 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456619 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.456839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.456713 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585313 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585463 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nbc\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585542 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585615 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585655 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585719 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585852 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585895 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.589149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.585982 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.590603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.590554 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.591910 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.591888 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.594869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.593665 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.595027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.593927 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.595123 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.594180 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.622924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.622870 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.651330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.651283 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.675820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.665608 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nbc\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.710743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.707411 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\") pod \"e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:16.904504 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:16.904403 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:17.365824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.363325 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.365824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.363390 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:17.403356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.403321 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:26:17.429536 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:17.429504 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9050aba_2a56_4fe3_8cdf_7abe2696bc74.slice/crio-f3046cce3a9fdc60decaa8beaca9a9ec648b6b4d84d6369053024a12cdd96343 WatchSource:0}: Error finding container f3046cce3a9fdc60decaa8beaca9a9ec648b6b4d84d6369053024a12cdd96343: Status 404 returned error can't find the container with id f3046cce3a9fdc60decaa8beaca9a9ec648b6b4d84d6369053024a12cdd96343 Oct 30 22:26:17.432698 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.432663 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:17.432791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.432753 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:17.432854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.432810 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:17.922206 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.922166 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerStarted","Data":"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c"} Oct 30 22:26:17.922360 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:17.922231 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerStarted","Data":"f3046cce3a9fdc60decaa8beaca9a9ec648b6b4d84d6369053024a12cdd96343"} Oct 30 22:26:18.044382 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.044331 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podStartSLOduration=2.044317018 podStartE2EDuration="2.044317018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:18.040671351 +0000 UTC m=+1498.635759699" watchObservedRunningTime="2026-10-30 22:26:18.044317018 +0000 UTC m=+1498.639405325" Oct 30 22:26:18.360641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.360178 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.360641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.360242 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:18.905454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.905420 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:18.911622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.911593 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:18.929266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.929238 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:18.929922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:18.929899 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:26:19.367821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:19.357468 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.367821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:19.357529 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:19.717066 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:19.716936 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:26:19.718776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:19.718749 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:26:20.370914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:20.367303 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:20.370914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:20.367367 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:21.361821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:21.360156 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:21.361821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:21.360218 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:22.353151 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:22.353113 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:22.353536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:22.353181 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:23.352993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:23.352957 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:26:23.353153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:23.353030 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 22:26:25.369406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:25.369368 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:25.369917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:25.369429 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:25.403082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:25.403034 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:25.403499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:25.403477 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:25.404327 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:25.404303 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:26:37.486221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.486127 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:26:37.492711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.492688 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.542036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.542000 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cc8c7d75-serving-cert\"" Oct 30 22:26:37.592569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.592526 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:26:37.608183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.608139 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.608443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.608395 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.608638 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.608538 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.608638 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.608584 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.709701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.709657 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.709912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.709824 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.709912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.709888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.710039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.709961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.710161 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:37.710136 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-serving-cert: secret "maas-api-e2e-migrate-cc8c7d75-serving-cert" not found Oct 30 22:26:37.710251 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:37.710231 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls podName:5c7e077f-9122-432a-89e8-55dd93f2454d nodeName:}" failed. No retries permitted until 2026-10-30 22:26:38.210208907 +0000 UTC m=+1518.805297195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls") pod "maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" (UID: "5c7e077f-9122-432a-89e8-55dd93f2454d") : secret "maas-api-e2e-migrate-cc8c7d75-serving-cert" not found Oct 30 22:26:37.710477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.710454 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.712779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.712675 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:37.805877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:37.804836 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:38.226279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:38.226243 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:38.228890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:38.228860 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") pod \"maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:38.408145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:38.408105 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:38.791478 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:38.791434 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7e077f_9122_432a_89e8_55dd93f2454d.slice/crio-96b31ec29bb00a753a1f0639af9849ff74169264dd4ef664583c2de218ee991f WatchSource:0}: Error finding container 96b31ec29bb00a753a1f0639af9849ff74169264dd4ef664583c2de218ee991f: Status 404 returned error can't find the container with id 96b31ec29bb00a753a1f0639af9849ff74169264dd4ef664583c2de218ee991f Oct 30 22:26:38.816008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:38.815980 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:26:39.239765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:39.239726 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" event={"ID":"5c7e077f-9122-432a-89e8-55dd93f2454d","Type":"ContainerStarted","Data":"96b31ec29bb00a753a1f0639af9849ff74169264dd4ef664583c2de218ee991f"} Oct 30 22:26:40.247603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:40.247560 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" event={"ID":"5c7e077f-9122-432a-89e8-55dd93f2454d","Type":"ContainerStarted","Data":"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87"} Oct 30 22:26:40.248021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:40.247649 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:40.362092 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:40.362028 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" podStartSLOduration=2.392140635 podStartE2EDuration="3.362008016s" totalImagesPullingTime="969.867381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:40.358336806 +0000 UTC m=+1520.953425112" watchObservedRunningTime="2026-10-30 22:26:40.362008016 +0000 UTC m=+1520.957096323" Oct 30 22:26:46.110693 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:46.110659 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:26:46.259617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.259587 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:26:46.546178 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.546140 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:26:46.550992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.550971 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.600838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.600810 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e8ed1149-serving-cert\"" Oct 30 22:26:46.609714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.609681 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:26:46.610671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.610641 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.610844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.610740 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.611057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.611034 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.611185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.611167 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xf2\" (UniqueName: \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.713139 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.713107 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.713337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.713188 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.713337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.713286 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.713471 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:46.713342 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-serving-cert: secret "maas-api-e2e-derive-e8ed1149-serving-cert" not found Oct 30 22:26:46.713471 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.713394 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xf2\" (UniqueName: \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.713471 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:46.713448 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls podName:fbb2493d-c47e-4862-a2d0-912761c9f659 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:47.21342655 +0000 UTC m=+1527.808514833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls") pod "maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" (UID: "fbb2493d-c47e-4862-a2d0-912761c9f659") : secret "maas-api-e2e-derive-e8ed1149-serving-cert" not found Oct 30 22:26:46.713901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.713874 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.716104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.716080 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:46.771700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:46.771669 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xf2\" (UniqueName: \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:47.228594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:47.228561 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:47.231059 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:47.231030 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") pod \"maas-api-e2e-derive-e8ed1149-66877877d9-vpf97\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:47.465070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:47.465039 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:47.858111 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:47.858027 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:26:47.862086 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:47.862059 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb2493d_c47e_4862_a2d0_912761c9f659.slice/crio-2dfb1332d8ede3b31ef3251ef1fe3e6fdcc12ad9e12cc58d4736f7a5d6a1b1ab WatchSource:0}: Error finding container 2dfb1332d8ede3b31ef3251ef1fe3e6fdcc12ad9e12cc58d4736f7a5d6a1b1ab: Status 404 returned error can't find the container with id 2dfb1332d8ede3b31ef3251ef1fe3e6fdcc12ad9e12cc58d4736f7a5d6a1b1ab Oct 30 22:26:48.335959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:48.335911 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" event={"ID":"fbb2493d-c47e-4862-a2d0-912761c9f659","Type":"ContainerStarted","Data":"2dfb1332d8ede3b31ef3251ef1fe3e6fdcc12ad9e12cc58d4736f7a5d6a1b1ab"} Oct 30 22:26:49.428871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:49.428762 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" event={"ID":"fbb2493d-c47e-4862-a2d0-912761c9f659","Type":"ContainerStarted","Data":"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5"} Oct 30 22:26:49.428871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:49.428862 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:55.488126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:55.488092 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:26:55.602943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:55.602892 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" podStartSLOduration=8.712418723 podStartE2EDuration="9.602877905s" totalImagesPullingTime="890.459182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:49.576172811 +0000 UTC m=+1530.171261117" watchObservedRunningTime="2026-10-30 22:26:55.602877905 +0000 UTC m=+1536.197966210" Oct 30 22:26:56.292696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.292664 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:26:56.297472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.297452 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.356852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.356825 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-926fc6-serving-cert\"" Oct 30 22:26:56.379750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.379719 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9vpk\" (UniqueName: \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.379917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.379788 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.380048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.380018 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.380155 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.380136 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.400423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.400396 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:26:56.481889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.481850 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9vpk\" (UniqueName: \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.482091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.481910 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.482091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.482064 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.482228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.482140 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.482485 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:56.482454 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-serving-cert: secret "maas-api-e2e-shared-a-w2-926fc6-serving-cert" not found Oct 30 22:26:56.482612 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:26:56.482537 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls podName:112fece8-9f56-4481-a69e-6bc3b47baf0d nodeName:}" failed. No retries permitted until 2026-10-30 22:26:56.982517255 +0000 UTC m=+1537.577605559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" (UID: "112fece8-9f56-4481-a69e-6bc3b47baf0d") : secret "maas-api-e2e-shared-a-w2-926fc6-serving-cert" not found Oct 30 22:26:56.482949 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.482927 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.485275 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.485251 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:56.542672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:56.542632 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9vpk\" (UniqueName: \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:57.015821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:57.012811 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:57.156824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:57.151742 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:57.255826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:57.252010 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:57.806867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:57.806828 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:26:57.811723 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:26:57.811693 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112fece8_9f56_4481_a69e_6bc3b47baf0d.slice/crio-51b0fab3c5dce2611a7e1ed062c954442085b8cd161aba08572a87ea45ccb3a6 WatchSource:0}: Error finding container 51b0fab3c5dce2611a7e1ed062c954442085b8cd161aba08572a87ea45ccb3a6: Status 404 returned error can't find the container with id 51b0fab3c5dce2611a7e1ed062c954442085b8cd161aba08572a87ea45ccb3a6 Oct 30 22:26:58.539271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:58.539237 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" event={"ID":"112fece8-9f56-4481-a69e-6bc3b47baf0d","Type":"ContainerStarted","Data":"51b0fab3c5dce2611a7e1ed062c954442085b8cd161aba08572a87ea45ccb3a6"} Oct 30 22:26:59.610594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:59.610552 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" event={"ID":"112fece8-9f56-4481-a69e-6bc3b47baf0d","Type":"ContainerStarted","Data":"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b"} Oct 30 22:26:59.617841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:59.617784 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:26:59.732227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:26:59.732178 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" podStartSLOduration=2.800576919 podStartE2EDuration="3.732163672s" totalImagesPullingTime="931.586753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:59.728925024 +0000 UTC m=+1540.324013333" watchObservedRunningTime="2026-10-30 22:26:59.732163672 +0000 UTC m=+1540.327252019" Oct 30 22:27:05.622084 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:05.622052 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:27:16.124582 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:16.124541 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:27:16.938718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:16.938677 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:27:16.943843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:16.943823 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.000765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:16.993472 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-7c5d18-serving-cert\"" Oct 30 22:27:17.008038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.008007 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:27:17.018538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.018500 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.018712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.018693 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9th\" (UniqueName: \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.018856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.018817 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.018939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.018884 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.120528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.120485 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.120862 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:17.120701 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-serving-cert: secret "maas-api-e2e-shared-a-w5-7c5d18-serving-cert" not found Oct 30 22:27:17.120962 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:17.120947 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls podName:e68995b7-5379-48de-b2f2-376b8196a77a nodeName:}" failed. No retries permitted until 2026-10-30 22:27:17.620921664 +0000 UTC m=+1558.216009949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls") pod "maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" (UID: "e68995b7-5379-48de-b2f2-376b8196a77a") : secret "maas-api-e2e-shared-a-w5-7c5d18-serving-cert" not found Oct 30 22:27:17.121208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.121170 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9th\" (UniqueName: \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.121435 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.121416 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.121615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.121596 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.122622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.122599 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.127825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.126995 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.269883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.261697 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9th\" (UniqueName: \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.641505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.640461 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.739228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.711383 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:17.908593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:17.908512 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:18.338672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:18.338630 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:27:18.923652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:18.923619 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" event={"ID":"e68995b7-5379-48de-b2f2-376b8196a77a","Type":"ContainerStarted","Data":"ad9b6e0f7db34a13b75ea8a6c68ad21e80e6a3e2d63e7b4bb919f40253e6fcfc"} Oct 30 22:27:19.931536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:19.931502 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" event={"ID":"e68995b7-5379-48de-b2f2-376b8196a77a","Type":"ContainerStarted","Data":"f81d8afd277211f629e56e625f310e4b9b245afa791b03e68fb989e24bddd1af"} Oct 30 22:27:19.931964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:19.931584 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:20.064822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:20.054316 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" podStartSLOduration=2.971448888 podStartE2EDuration="4.05429733s" totalImagesPullingTime="1.082848442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:20.041777037 +0000 UTC m=+1560.636865343" watchObservedRunningTime="2026-10-30 22:27:20.05429733 +0000 UTC m=+1560.649385637" Oct 30 22:27:25.943751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.943718 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:27:25.981984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.981954 4305 generic.go:356] "Generic (PLEG): container finished" podID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerID="0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba" exitCode=0 Oct 30 22:27:25.982814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.982020 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerDied","Data":"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba"} Oct 30 22:27:25.982814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.982448 4305 scope.go:122] "RemoveContainer" containerID="0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba" Oct 30 22:27:25.984470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.984429 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:25.984547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.984531 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:25.984592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:25.984578 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:27.030488 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.030451 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerStarted","Data":"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1"} Oct 30 22:27:27.036729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.036682 4305 generic.go:356] "Generic (PLEG): container finished" podID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerID="9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac" exitCode=0 Oct 30 22:27:27.036928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.036750 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerDied","Data":"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac"} Oct 30 22:27:27.037359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.037333 4305 scope.go:122] "RemoveContainer" containerID="9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac" Oct 30 22:27:27.039539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.039492 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:27.039647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.039621 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:27.040190 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.040133 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:27.385531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.385411 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:27:27.385975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.385951 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:27:27.386199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:27.386155 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:27:28.053824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.046506 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerStarted","Data":"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7"} Oct 30 22:27:28.383459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.383359 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:27:28.383722 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.383690 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:27:28.472822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.472176 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:27:28.476819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.473786 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:28.476819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:28.473867 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:29.387308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:29.387264 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:27:29.387979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:29.387942 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:27:29.466928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:29.466891 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:29.467221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:29.467185 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:30.385943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:30.385904 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:27:30.386425 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:30.386394 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:27:30.472981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:30.472944 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:30.473572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:30.473542 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:31.396037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:31.395992 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:27:31.396240 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:31.396094 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:27:31.463819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:31.462164 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:31.463819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:31.462237 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:31.764941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:31.764900 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:27:32.430920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:32.430885 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:27:32.432475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:32.432450 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:27:32.433949 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:32.433917 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:27:32.463223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:32.463192 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:32.463502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:32.463476 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:33.461920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:33.461881 4305 patch_prober.go:29] interesting pod/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:27:33.462369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:33.461951 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:27:34.466975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:34.466940 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:27:34.468173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:34.468146 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:27:46.111895 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:46.111857 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:27:50.406434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.406396 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:27:50.411235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.411214 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.480535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.480498 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:27:50.504251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.504215 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.504451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.504286 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.504451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.504324 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf7v\" (UniqueName: \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.504451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.504384 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.521072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.521040 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-29bbe735-serving-cert\"" Oct 30 22:27:50.606965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.606921 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.607181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.607003 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.607181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.607052 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf7v\" (UniqueName: \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.607181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.607122 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.608001 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:50.607968 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-serving-cert: secret "maas-api-e2e-mt-29bbe735-serving-cert" not found Oct 30 22:27:50.608156 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:27:50.608051 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls podName:adefb7d0-117f-40a1-957b-e7b109de05d6 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:51.10802987 +0000 UTC m=+1591.703118157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls") pod "maas-api-e2e-mt-29bbe735-7948869679-vnl6n" (UID: "adefb7d0-117f-40a1-957b-e7b109de05d6") : secret "maas-api-e2e-mt-29bbe735-serving-cert" not found Oct 30 22:27:50.608156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.608095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.609982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.609953 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:50.770820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:50.765745 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf7v\" (UniqueName: \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:51.137836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:51.137712 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:51.140112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:51.140086 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-vnl6n\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:51.326622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:51.326589 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:51.740242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:51.740211 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:27:51.742163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:27:51.742135 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadefb7d0_117f_40a1_957b_e7b109de05d6.slice/crio-8f56989a21b28fd8352ac349ffb15a432a11d1efae40ddf77a02e61672078f71 WatchSource:0}: Error finding container 8f56989a21b28fd8352ac349ffb15a432a11d1efae40ddf77a02e61672078f71: Status 404 returned error can't find the container with id 8f56989a21b28fd8352ac349ffb15a432a11d1efae40ddf77a02e61672078f71 Oct 30 22:27:52.309210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:52.309175 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" event={"ID":"adefb7d0-117f-40a1-957b-e7b109de05d6","Type":"ContainerStarted","Data":"8f56989a21b28fd8352ac349ffb15a432a11d1efae40ddf77a02e61672078f71"} Oct 30 22:27:53.331168 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:53.331123 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" event={"ID":"adefb7d0-117f-40a1-957b-e7b109de05d6","Type":"ContainerStarted","Data":"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201"} Oct 30 22:27:53.331663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:53.331350 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:27:53.473868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:53.473786 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" podStartSLOduration=2.454859885 podStartE2EDuration="3.473765587s" totalImagesPullingTime="1.018905702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:53.471361374 +0000 UTC m=+1594.066449716" watchObservedRunningTime="2026-10-30 22:27:53.473765587 +0000 UTC m=+1594.068853892" Oct 30 22:27:59.437779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:27:59.437659 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:28:00.443874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.443845 4305 generic.go:356] "Generic (PLEG): container finished" podID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerID="05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c" exitCode=0 Oct 30 22:28:00.444300 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.443925 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerDied","Data":"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c"} Oct 30 22:28:00.444434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.444419 4305 scope.go:122] "RemoveContainer" containerID="05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c" Oct 30 22:28:00.446260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.446227 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:28:00.446371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.446316 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:28:00.446439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:00.446368 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:28:01.472653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:01.472595 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerStarted","Data":"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e"} Oct 30 22:28:01.905604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:01.905517 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:28:01.906686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:01.906657 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:28:01.906840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:01.906707 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 22:28:02.905282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:02.905246 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:28:02.905675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:02.905313 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 22:28:03.905536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:03.905500 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:28:03.906071 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:03.905562 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 22:28:03.930122 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:03.930091 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:28:04.905148 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:04.905105 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:28:04.905378 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:04.905219 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 22:28:05.905329 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:05.905280 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:28:05.905718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:05.905347 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 22:28:06.967364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:06.967318 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:28:06.969401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:06.969368 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:28:16.118751 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:16.118713 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:28:45.555612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:45.555577 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:28:45.556211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:45.555768 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" podUID="5c7e077f-9122-432a-89e8-55dd93f2454d" containerName="maas-api" containerID="cri-o://7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87" gracePeriod=30 Oct 30 22:28:46.134870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.131374 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:28:46.258689 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:46.257954 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:28:46.272017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.271989 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\") pod \"5c7e077f-9122-432a-89e8-55dd93f2454d\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " Oct 30 22:28:46.272251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.272030 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\") pod \"5c7e077f-9122-432a-89e8-55dd93f2454d\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " Oct 30 22:28:46.272251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.272109 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") pod \"5c7e077f-9122-432a-89e8-55dd93f2454d\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " Oct 30 22:28:46.272251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.272215 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\") pod \"5c7e077f-9122-432a-89e8-55dd93f2454d\" (UID: \"5c7e077f-9122-432a-89e8-55dd93f2454d\") " Oct 30 22:28:46.272498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.272411 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca" pod "5c7e077f-9122-432a-89e8-55dd93f2454d" (UID: "5c7e077f-9122-432a-89e8-55dd93f2454d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:46.272898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.272878 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7e077f-9122-432a-89e8-55dd93f2454d-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:46.274787 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.274763 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls" pod "5c7e077f-9122-432a-89e8-55dd93f2454d" (UID: "5c7e077f-9122-432a-89e8-55dd93f2454d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:46.274907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.274856 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl" pod "5c7e077f-9122-432a-89e8-55dd93f2454d" (UID: "5c7e077f-9122-432a-89e8-55dd93f2454d"). InnerVolumeSpecName "kube-api-access-nj2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:46.281147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.281120 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs" pod "5c7e077f-9122-432a-89e8-55dd93f2454d" (UID: "5c7e077f-9122-432a-89e8-55dd93f2454d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:46.380218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.380152 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:46.380218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.380200 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7e077f-9122-432a-89e8-55dd93f2454d-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:46.380504 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.380298 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2vl\" (UniqueName: \"kubernetes.io/projected/5c7e077f-9122-432a-89e8-55dd93f2454d-kube-api-access-nj2vl\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:46.445146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.445114 4305 generic.go:356] "Generic (PLEG): container finished" podID="5c7e077f-9122-432a-89e8-55dd93f2454d" containerID="7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87" exitCode=0 Oct 30 22:28:46.445316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.445159 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" Oct 30 22:28:46.445316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.445158 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" event={"ID":"5c7e077f-9122-432a-89e8-55dd93f2454d","Type":"ContainerDied","Data":"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87"} Oct 30 22:28:46.445316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.445292 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt" event={"ID":"5c7e077f-9122-432a-89e8-55dd93f2454d","Type":"ContainerDied","Data":"96b31ec29bb00a753a1f0639af9849ff74169264dd4ef664583c2de218ee991f"} Oct 30 22:28:46.445432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.445331 4305 scope.go:122] "RemoveContainer" containerID="7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87" Oct 30 22:28:46.456314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.456298 4305 scope.go:122] "RemoveContainer" containerID="7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87" Oct 30 22:28:46.456542 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:46.456522 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87\": container with ID starting with 7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87 not found: ID does not exist" containerID="7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87" Oct 30 22:28:46.456587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.456551 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87"} err="failed to get container status \"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87\": rpc error: code = NotFound desc = could not find container \"7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87\": container with ID starting with 7de8682cc3fe30421ddda7c5d66bd8a724cfaa36af2e5364ec9ddc52d4b03f87 not found: ID does not exist" Oct 30 22:28:46.617642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.617610 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:28:46.621612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:46.621588 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cc8c7d75-7d869546c8-dgvwt"] Oct 30 22:28:47.547658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:47.547622 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7e077f-9122-432a-89e8-55dd93f2454d" path="/var/lib/kubelet/pods/5c7e077f-9122-432a-89e8-55dd93f2454d/volumes" Oct 30 22:28:51.759195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:51.759160 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:28:51.759703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:51.759358 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" podUID="fbb2493d-c47e-4862-a2d0-912761c9f659" containerName="maas-api" containerID="cri-o://a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5" gracePeriod=30 Oct 30 22:28:52.408250 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.408212 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:28:52.464824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.464241 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\" (UniqueName: \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\") pod \"fbb2493d-c47e-4862-a2d0-912761c9f659\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " Oct 30 22:28:52.464824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.464415 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") pod \"fbb2493d-c47e-4862-a2d0-912761c9f659\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " Oct 30 22:28:52.464824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.464549 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\") pod \"fbb2493d-c47e-4862-a2d0-912761c9f659\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " Oct 30 22:28:52.464824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.464671 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\") pod \"fbb2493d-c47e-4862-a2d0-912761c9f659\" (UID: \"fbb2493d-c47e-4862-a2d0-912761c9f659\") " Oct 30 22:28:52.469220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.469181 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca" pod "fbb2493d-c47e-4862-a2d0-912761c9f659" (UID: "fbb2493d-c47e-4862-a2d0-912761c9f659"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:52.476752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.475428 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs" pod "fbb2493d-c47e-4862-a2d0-912761c9f659" (UID: "fbb2493d-c47e-4862-a2d0-912761c9f659"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:52.476752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.476569 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls" pod "fbb2493d-c47e-4862-a2d0-912761c9f659" (UID: "fbb2493d-c47e-4862-a2d0-912761c9f659"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:52.476752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.476698 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2" pod "fbb2493d-c47e-4862-a2d0-912761c9f659" (UID: "fbb2493d-c47e-4862-a2d0-912761c9f659"). InnerVolumeSpecName "kube-api-access-g7xf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:52.568055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.568019 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:52.568055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.568052 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xf2\" (UniqueName: \"kubernetes.io/projected/fbb2493d-c47e-4862-a2d0-912761c9f659-kube-api-access-g7xf2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:52.568055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.568063 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb2493d-c47e-4862-a2d0-912761c9f659-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:52.568296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.568073 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb2493d-c47e-4862-a2d0-912761c9f659-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:52.596550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.596510 4305 generic.go:356] "Generic (PLEG): container finished" podID="fbb2493d-c47e-4862-a2d0-912761c9f659" containerID="a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5" exitCode=0 Oct 30 22:28:52.596728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.596565 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" Oct 30 22:28:52.596728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.596605 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" event={"ID":"fbb2493d-c47e-4862-a2d0-912761c9f659","Type":"ContainerDied","Data":"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5"} Oct 30 22:28:52.596728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.596680 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97" event={"ID":"fbb2493d-c47e-4862-a2d0-912761c9f659","Type":"ContainerDied","Data":"2dfb1332d8ede3b31ef3251ef1fe3e6fdcc12ad9e12cc58d4736f7a5d6a1b1ab"} Oct 30 22:28:52.596728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.596716 4305 scope.go:122] "RemoveContainer" containerID="a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5" Oct 30 22:28:52.609154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.609135 4305 scope.go:122] "RemoveContainer" containerID="a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5" Oct 30 22:28:52.609467 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:52.609451 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5\": container with ID starting with a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5 not found: ID does not exist" containerID="a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5" Oct 30 22:28:52.609524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.609476 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5"} err="failed to get container status \"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5\": rpc error: code = NotFound desc = could not find container \"a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5\": container with ID starting with a8f7b41a813e83ddd98fdaa2c2a11254b45824c07bc857d7a0c598bf846f38e5 not found: ID does not exist" Oct 30 22:28:52.807552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.807519 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:28:52.882821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:52.882052 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e8ed1149-66877877d9-vpf97"] Oct 30 22:28:53.520657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:53.520620 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb2493d-c47e-4862-a2d0-912761c9f659" path="/var/lib/kubelet/pods/fbb2493d-c47e-4862-a2d0-912761c9f659/volumes" Oct 30 22:28:54.176007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:54.175978 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:28:54.176474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:54.176218 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" containerID="cri-o://80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7" gracePeriod=30 Oct 30 22:28:59.464231 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.464206 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:28:59.490876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.490832 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.490887 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.490949 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491001 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491036 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491125 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491223 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:59.491361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491311 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491371 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491398 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\") pod \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\" (UID: \"6d5b82dc-26ae-4412-8482-bef18cfb4e6e\") " Oct 30 22:28:59.491612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491585 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:59.491701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491608 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:59.491701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491595 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:59.491701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.491665 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:59.492137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.492116 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.492137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.492137 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.492290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.492146 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.492290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.492155 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.492290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.492168 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.493297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.493270 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:59.493424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.493401 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "kube-api-access-hmcw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:59.493505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.493489 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:59.493779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.493765 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token" pod "6d5b82dc-26ae-4412-8482-bef18cfb4e6e" (UID: "6d5b82dc-26ae-4412-8482-bef18cfb4e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:59.593856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.593829 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.593856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.593853 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.594046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.593863 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.594046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.593875 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcw8\" (UniqueName: \"kubernetes.io/projected/6d5b82dc-26ae-4412-8482-bef18cfb4e6e-kube-api-access-hmcw8\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:59.785883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.784105 4305 generic.go:356] "Generic (PLEG): container finished" podID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerID="80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7" exitCode=0 Oct 30 22:28:59.785883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.784471 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerDied","Data":"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7"} Oct 30 22:28:59.785883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.784550 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" event={"ID":"6d5b82dc-26ae-4412-8482-bef18cfb4e6e","Type":"ContainerDied","Data":"d169984cd9d5acc1428503e16cdbfc4daf6bac623e788b88bb7d249e8fbbd53e"} Oct 30 22:28:59.785883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.784583 4305 scope.go:122] "RemoveContainer" containerID="80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7" Oct 30 22:28:59.785883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.784849 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws" Oct 30 22:28:59.883116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.879038 4305 scope.go:122] "RemoveContainer" containerID="9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac" Oct 30 22:28:59.904827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.904788 4305 scope.go:122] "RemoveContainer" containerID="80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7" Oct 30 22:28:59.905480 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:59.905452 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7\": container with ID starting with 80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7 not found: ID does not exist" containerID="80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7" Oct 30 22:28:59.905636 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.905613 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7"} err="failed to get container status \"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7\": rpc error: code = NotFound desc = could not find container \"80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7\": container with ID starting with 80c0594549af0efdd58709bfac676c158d0c8659aa093b67e65cfc149ce604f7 not found: ID does not exist" Oct 30 22:28:59.905728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.905718 4305 scope.go:122] "RemoveContainer" containerID="9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac" Oct 30 22:28:59.906198 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:28:59.906177 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac\": container with ID starting with 9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac not found: ID does not exist" containerID="9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac" Oct 30 22:28:59.906337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.906315 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac"} err="failed to get container status \"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac\": rpc error: code = NotFound desc = could not find container \"9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac\": container with ID starting with 9307703f8ff676758cf1ca69f0c7fbe243bed7340390723916f2d950f5a2beac not found: ID does not exist" Oct 30 22:28:59.966821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.966777 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:28:59.972109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:28:59.972087 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e8ed1149-openshift-default-ff9d86fc-pccws"] Oct 30 22:29:01.519742 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:01.519709 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" path="/var/lib/kubelet/pods/6d5b82dc-26ae-4412-8482-bef18cfb4e6e/volumes" Oct 30 22:29:05.733271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.733229 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:29:05.734164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734141 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5c7e077f-9122-432a-89e8-55dd93f2454d" containerName="maas-api" Oct 30 22:29:05.734164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734165 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7e077f-9122-432a-89e8-55dd93f2454d" containerName="maas-api" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734203 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fbb2493d-c47e-4862-a2d0-912761c9f659" containerName="maas-api" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734212 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb2493d-c47e-4862-a2d0-912761c9f659" containerName="maas-api" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734231 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734238 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734289 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734297 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734451 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb2493d-c47e-4862-a2d0-912761c9f659" containerName="maas-api" Oct 30 22:29:05.734618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734465 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734483 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5b82dc-26ae-4412-8482-bef18cfb4e6e" containerName="istio-proxy" Oct 30 22:29:05.734618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.734494 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7e077f-9122-432a-89e8-55dd93f2454d" containerName="maas-api" Oct 30 22:29:05.739642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.739618 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.793044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.792999 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:29:05.839998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.839966 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-29bbe735\"/\"kube-root-ca.crt\"" Oct 30 22:29:05.840295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.840083 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-29bbe735\"/\"default-dockercfg-2cwqb\"" Oct 30 22:29:05.840295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.840159 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-29bbe735\"/\"openshift-service-ca.crt\"" Oct 30 22:29:05.845708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.845687 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-29bbe735\"/\"e2e-lifecycle-model-29bbe735-kserve-self-signed-certs\"" Oct 30 22:29:05.912823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.910234 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.915835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.915781 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.924977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.916300 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.925558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.925513 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j7d\" (UniqueName: \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.925873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.925854 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:05.951824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:05.943984 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060119 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060171 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060293 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060406 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j7d\" (UniqueName: \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.060744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.060646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.064515 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.064403 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.066067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.065643 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.066067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.065662 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.066067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.065949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.067934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.067899 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.125135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.125103 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j7d\" (UniqueName: \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\") pod \"e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.353977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.353887 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:06.828475 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:29:06.828444 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb2a136_901d_4936_9c35_fadb3df7307d.slice/crio-47e466aeb4bbdcf2465a38bb12b2bdd655a65393ebd858a47ced46f13b09ee78 WatchSource:0}: Error finding container 47e466aeb4bbdcf2465a38bb12b2bdd655a65393ebd858a47ced46f13b09ee78: Status 404 returned error can't find the container with id 47e466aeb4bbdcf2465a38bb12b2bdd655a65393ebd858a47ced46f13b09ee78 Oct 30 22:29:06.830489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:06.830466 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:29:07.261560 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.261524 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" event={"ID":"ecb2a136-901d-4936-9c35-fadb3df7307d","Type":"ContainerStarted","Data":"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7"} Oct 30 22:29:07.261740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.261568 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" event={"ID":"ecb2a136-901d-4936-9c35-fadb3df7307d","Type":"ContainerStarted","Data":"47e466aeb4bbdcf2465a38bb12b2bdd655a65393ebd858a47ced46f13b09ee78"} Oct 30 22:29:07.354502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.354465 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:07.386594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.386531 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" podStartSLOduration=2.386511295 podStartE2EDuration="2.386511295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:07.383396746 +0000 UTC m=+1667.978485062" watchObservedRunningTime="2026-10-30 22:29:07.386511295 +0000 UTC m=+1667.981599606" Oct 30 22:29:07.445240 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.445209 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:29:07.445441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.445418 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" containerID="cri-o://235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776" gracePeriod=30 Oct 30 22:29:07.449465 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.449437 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:29:07.449665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.449642 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" containerID="cri-o://7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd" gracePeriod=30 Oct 30 22:29:07.453408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.453382 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:29:07.453622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:07.453595 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" containerID="cri-o://6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc" gracePeriod=30 Oct 30 22:29:08.230106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.108350 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:29:08.230106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.108620 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" containerID="cri-o://e87e41739019f271082b8cff313e41466ed29841bc5f7d7c64f6e0342bbd47b6" gracePeriod=30 Oct 30 22:29:08.403814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.403754 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:29:08.404266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.404230 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" containerID="cri-o://99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1" gracePeriod=30 Oct 30 22:29:08.410782 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.410750 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:29:08.411191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.411137 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" containerID="cri-o://efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54" gracePeriod=30 Oct 30 22:29:08.414512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.414485 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:08.414618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.414561 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:08.973118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.973071 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:29:08.973400 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.973362 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" containerID="cri-o://46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d" gracePeriod=30 Oct 30 22:29:08.979897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.979868 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:29:08.980169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.980118 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" containerID="cri-o://735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f" gracePeriod=30 Oct 30 22:29:08.982650 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.982626 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:29:08.982888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:08.982862 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" containerID="cri-o://f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae" gracePeriod=30 Oct 30 22:29:12.608787 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.608736 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:29:12.609286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.608844 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 22:29:12.762041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.762007 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:29:12.808067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.808039 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:29:12.832912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.832885 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:29:12.859351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859316 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859430 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859490 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859522 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859559 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.859637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859577 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859678 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859739 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859807 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.859997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859897 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.859997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859914 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.859997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.859962 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\") pod \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\" (UID: \"e65fccc6-fd3f-4757-b5bb-bd2254834e73\") " Oct 30 22:29:12.860354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860327 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.860513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860397 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:12.860857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860824 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.860857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860854 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.861041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860868 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.861041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860883 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.861041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.860897 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.862056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.862027 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.862239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.862211 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:12.862372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.862345 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "kube-api-access-wkskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.862449 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.862428 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token" pod "e65fccc6-fd3f-4757-b5bb-bd2254834e73" (UID: "e65fccc6-fd3f-4757-b5bb-bd2254834e73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.961740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961707 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961761 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961817 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961854 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961882 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961963 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.961990 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962110 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962269 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962319 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962367 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962441 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962471 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962510 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962521 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962589 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962713 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\") pod \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\" (UID: \"a9b78c95-1ae0-4e46-83f7-7217547a25ed\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962756 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\") pod \"7f20bbd8-30b8-419c-8534-4c83e76665be\" (UID: \"7f20bbd8-30b8-419c-8534-4c83e76665be\") " Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963728 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963753 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963773 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963789 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkskj\" (UniqueName: \"kubernetes.io/projected/e65fccc6-fd3f-4757-b5bb-bd2254834e73-kube-api-access-wkskj\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963818 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65fccc6-fd3f-4757-b5bb-bd2254834e73-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.962756 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963196 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963539 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.963875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.964036 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.964209 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.964429 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.965731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.965100 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:12.967533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.966044 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.970540 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.970508 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.970870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.970836 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:12.971034 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971006 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:12.971136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971116 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "kube-api-access-p8ncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.971197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971124 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:12.971444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971427 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626" pod "7f20bbd8-30b8-419c-8534-4c83e76665be" (UID: "7f20bbd8-30b8-419c-8534-4c83e76665be"). InnerVolumeSpecName "kube-api-access-fl626". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.971734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971710 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:12.971881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:12.971866 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy" pod "a9b78c95-1ae0-4e46-83f7-7217547a25ed" (UID: "a9b78c95-1ae0-4e46-83f7-7217547a25ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067713 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f20bbd8-30b8-419c-8534-4c83e76665be-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067746 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ncb\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-kube-api-access-p8ncb\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067756 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067766 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067775 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067785 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067809 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.067830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067824 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl626\" (UniqueName: \"kubernetes.io/projected/7f20bbd8-30b8-419c-8534-4c83e76665be-kube-api-access-fl626\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067838 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067849 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067858 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067867 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067875 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067884 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067892 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f20bbd8-30b8-419c-8534-4c83e76665be-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067900 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.068219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.067909 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b78c95-1ae0-4e46-83f7-7217547a25ed-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.457105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.457073 4305 generic.go:356] "Generic (PLEG): container finished" podID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerID="6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc" exitCode=0 Oct 30 22:29:13.457220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.457132 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" Oct 30 22:29:13.457220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.457155 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" event={"ID":"a9b78c95-1ae0-4e46-83f7-7217547a25ed","Type":"ContainerDied","Data":"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc"} Oct 30 22:29:13.457335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.457229 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht" event={"ID":"a9b78c95-1ae0-4e46-83f7-7217547a25ed","Type":"ContainerDied","Data":"dac2bc89911bb36921eeba4eb0481b0e0c1fdb0176064bf52d36c8a645f0755f"} Oct 30 22:29:13.457335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.457253 4305 scope.go:122] "RemoveContainer" containerID="6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc" Oct 30 22:29:13.460096 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.460074 4305 generic.go:356] "Generic (PLEG): container finished" podID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerID="e87e41739019f271082b8cff313e41466ed29841bc5f7d7c64f6e0342bbd47b6" exitCode=0 Oct 30 22:29:13.460214 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.460145 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" event={"ID":"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b","Type":"ContainerDied","Data":"e87e41739019f271082b8cff313e41466ed29841bc5f7d7c64f6e0342bbd47b6"} Oct 30 22:29:13.462110 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.462088 4305 generic.go:356] "Generic (PLEG): container finished" podID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerID="7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd" exitCode=0 Oct 30 22:29:13.462623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.462145 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" Oct 30 22:29:13.462623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.462197 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" event={"ID":"7f20bbd8-30b8-419c-8534-4c83e76665be","Type":"ContainerDied","Data":"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd"} Oct 30 22:29:13.462623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.462247 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf" event={"ID":"7f20bbd8-30b8-419c-8534-4c83e76665be","Type":"ContainerDied","Data":"6415dbe8040902c793031230e2ef3db589cd260db1d00d19b00f37db6e98145b"} Oct 30 22:29:13.463900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.463874 4305 generic.go:356] "Generic (PLEG): container finished" podID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerID="235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776" exitCode=0 Oct 30 22:29:13.464095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.463907 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" event={"ID":"e65fccc6-fd3f-4757-b5bb-bd2254834e73","Type":"ContainerDied","Data":"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776"} Oct 30 22:29:13.464211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.464116 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" event={"ID":"e65fccc6-fd3f-4757-b5bb-bd2254834e73","Type":"ContainerDied","Data":"b500ae3bef1f79e2a8a11aafa46d58061c82c43923bee352bedc0e1a7a09d752"} Oct 30 22:29:13.464211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.463926 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2" Oct 30 22:29:13.584555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.584344 4305 scope.go:122] "RemoveContainer" containerID="6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc" Oct 30 22:29:13.584748 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:13.584728 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc\": container with ID starting with 6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc not found: ID does not exist" containerID="6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc" Oct 30 22:29:13.584817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.584761 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc"} err="failed to get container status \"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc\": rpc error: code = NotFound desc = could not find container \"6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc\": container with ID starting with 6b01e7a273fa06d10abf4ac3438a521a2a2cf2b1f02045aa4524fcb66e12e9fc not found: ID does not exist" Oct 30 22:29:13.584817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.584781 4305 scope.go:122] "RemoveContainer" containerID="7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd" Oct 30 22:29:13.598652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.598631 4305 scope.go:122] "RemoveContainer" containerID="7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd" Oct 30 22:29:13.599009 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:13.598985 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd\": container with ID starting with 7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd not found: ID does not exist" containerID="7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd" Oct 30 22:29:13.599081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.599015 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd"} err="failed to get container status \"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd\": rpc error: code = NotFound desc = could not find container \"7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd\": container with ID starting with 7a00e21c2b98cf07f4c0f77c5681136fb928c19d8c2b66a1a924d3d436500ecd not found: ID does not exist" Oct 30 22:29:13.599081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.599034 4305 scope.go:122] "RemoveContainer" containerID="235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776" Oct 30 22:29:13.615245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.614266 4305 scope.go:122] "RemoveContainer" containerID="235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776" Oct 30 22:29:13.616256 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:13.615756 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776\": container with ID starting with 235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776 not found: ID does not exist" containerID="235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776" Oct 30 22:29:13.616425 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.616384 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776"} err="failed to get container status \"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776\": rpc error: code = NotFound desc = could not find container \"235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776\": container with ID starting with 235a95bc6258816402c0b48802f4589b69562b5feeabfe32b1610b17235c7776 not found: ID does not exist" Oct 30 22:29:13.703324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.703297 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:29:13.745983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.745952 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:29:13.797652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.797621 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-cl8gf"] Oct 30 22:29:13.798280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798259 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798368 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798416 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798477 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798641 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798707 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798811 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798775 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798829 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.798947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798868 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\") pod \"109521bf-2196-4621-a52c-f0330c45f94d\" (UID: \"109521bf-2196-4621-a52c-f0330c45f94d\") " Oct 30 22:29:13.799926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798647 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.800044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.798879 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:13.800044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.799170 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.800044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.799885 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.800044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.799984 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.801051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.801023 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:13.801157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.801077 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "kube-api-access-6mlhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:13.801292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.801275 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:13.801597 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.801576 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo" pod "109521bf-2196-4621-a52c-f0330c45f94d" (UID: "109521bf-2196-4621-a52c-f0330c45f94d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:13.864225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.864193 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:29:13.868445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.868422 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-ptbht"] Oct 30 22:29:13.901480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901435 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901480 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901496 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/109521bf-2196-4621-a52c-f0330c45f94d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901515 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlhw\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-kube-api-access-6mlhw\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901530 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/109521bf-2196-4621-a52c-f0330c45f94d-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901543 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901558 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901572 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/109521bf-2196-4621-a52c-f0330c45f94d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.901710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.901585 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/109521bf-2196-4621-a52c-f0330c45f94d-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:13.997845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:13.997815 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:29:14.020909 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.020771 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-xfvw2"] Oct 30 22:29:14.156504 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.156470 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:29:14.166297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.166272 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:29:14.250450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250359 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.250608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250480 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.250608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250556 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.250608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250589 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.250751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.250751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250696 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.250894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250767 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.250951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250911 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.251007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.250959 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.251060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251036 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.251182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.251182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251150 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.251296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251206 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.251296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251239 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.251296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251272 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\") pod \"91068286-7de3-4d59-8d59-20180fe271c0\" (UID: \"91068286-7de3-4d59-8d59-20180fe271c0\") " Oct 30 22:29:14.251445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251296 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.251445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251334 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.251445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251401 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.251445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.251435 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\") pod \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\" (UID: \"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b\") " Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252122 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252339 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252365 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252339 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252532 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252642 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252733 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252921 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.253140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.252941 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.254002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.253966 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.254328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.254309 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.256269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.256251 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.257528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.257316 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.257528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.257339 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:14.257528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.257361 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "kube-api-access-wvql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.257528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.257424 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "kube-api-access-trcbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.257744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.257545 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo" pod "91068286-7de3-4d59-8d59-20180fe271c0" (UID: "91068286-7de3-4d59-8d59-20180fe271c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:14.258070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.258045 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.258070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.258057 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy" pod "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" (UID: "48e539ea-bd5d-4ab6-9b2f-8d93851fec5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.313031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.312994 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:29:14.354381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354340 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354381 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91068286-7de3-4d59-8d59-20180fe271c0-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354396 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354410 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354423 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354437 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354450 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354465 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354479 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trcbp\" (UniqueName: \"kubernetes.io/projected/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-kube-api-access-trcbp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354494 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354508 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354524 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91068286-7de3-4d59-8d59-20180fe271c0-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354537 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91068286-7de3-4d59-8d59-20180fe271c0-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354551 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvql4\" (UniqueName: \"kubernetes.io/projected/91068286-7de3-4d59-8d59-20180fe271c0-kube-api-access-wvql4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354564 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.354633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.354577 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.405188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.405098 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:29:14.411620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.411595 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:29:14.456107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456070 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456267 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456356 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456322 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456357 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456398 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456609 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456587 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456740 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.456836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.456781 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\") pod \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\" (UID: \"135ec54c-ac83-45ca-ae90-8f1e90855cc2\") " Oct 30 22:29:14.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.457220 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.457643 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.457970 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.458005 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.458226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.458036 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.458715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.458695 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.459323 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.459300 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:14.460531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.460497 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.460642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.460613 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "kube-api-access-pxd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.460914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.460895 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy" pod "135ec54c-ac83-45ca-ae90-8f1e90855cc2" (UID: "135ec54c-ac83-45ca-ae90-8f1e90855cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.477703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.477673 4305 generic.go:356] "Generic (PLEG): container finished" podID="109521bf-2196-4621-a52c-f0330c45f94d" containerID="99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1" exitCode=0 Oct 30 22:29:14.477830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.477730 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" Oct 30 22:29:14.477830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.477747 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" event={"ID":"109521bf-2196-4621-a52c-f0330c45f94d","Type":"ContainerDied","Data":"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1"} Oct 30 22:29:14.477994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.477874 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5" event={"ID":"109521bf-2196-4621-a52c-f0330c45f94d","Type":"ContainerDied","Data":"e54f0d7baef0533c4f430bcd950e1bd3c1c9b985a9d9a96f2171e6dbab8208e7"} Oct 30 22:29:14.477994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.477906 4305 scope.go:122] "RemoveContainer" containerID="99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1" Oct 30 22:29:14.482351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.480352 4305 generic.go:356] "Generic (PLEG): container finished" podID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerID="f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae" exitCode=0 Oct 30 22:29:14.482351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.480540 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" event={"ID":"9f7d9091-a50e-4d27-af57-e098d735a24e","Type":"ContainerDied","Data":"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae"} Oct 30 22:29:14.482351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.480582 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" event={"ID":"9f7d9091-a50e-4d27-af57-e098d735a24e","Type":"ContainerDied","Data":"1f0f005c4b7840dc585a431bf6cca37ced95e76ab08339d0dffabdcaad207feb"} Oct 30 22:29:14.482351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.480661 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp" Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.482666 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.482665 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c" event={"ID":"48e539ea-bd5d-4ab6-9b2f-8d93851fec5b","Type":"ContainerDied","Data":"17562b4e1694215cf0b7a94e9ef53881e19f60dc7ca46b1823ebc503d817e715"} Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.484741 4305 generic.go:356] "Generic (PLEG): container finished" podID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerID="735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f" exitCode=0 Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.484850 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" event={"ID":"c4ea0083-1fd5-4023-8c74-9d2978145014","Type":"ContainerDied","Data":"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f"} Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.484900 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" event={"ID":"c4ea0083-1fd5-4023-8c74-9d2978145014","Type":"ContainerDied","Data":"ff9090fc81ec63bd17ea784d8c98569f124e05275ac26d73d017cee28f77db7a"} Oct 30 22:29:14.485686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.484978 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj" Oct 30 22:29:14.487541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.487482 4305 generic.go:356] "Generic (PLEG): container finished" podID="91068286-7de3-4d59-8d59-20180fe271c0" containerID="efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54" exitCode=0 Oct 30 22:29:14.487773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.487707 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" event={"ID":"91068286-7de3-4d59-8d59-20180fe271c0","Type":"ContainerDied","Data":"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54"} Oct 30 22:29:14.488169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.488036 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" event={"ID":"91068286-7de3-4d59-8d59-20180fe271c0","Type":"ContainerDied","Data":"d3f3e8f0d4c7a1687c8f9f9365fa902524a25627db5de637f4dd1b60d7ac064c"} Oct 30 22:29:14.488169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.488121 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz" Oct 30 22:29:14.492431 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.492385 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" Oct 30 22:29:14.492534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.492491 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" event={"ID":"135ec54c-ac83-45ca-ae90-8f1e90855cc2","Type":"ContainerDied","Data":"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d"} Oct 30 22:29:14.496551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.492289 4305 generic.go:356] "Generic (PLEG): container finished" podID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerID="46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d" exitCode=0 Oct 30 22:29:14.496551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.492928 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs" event={"ID":"135ec54c-ac83-45ca-ae90-8f1e90855cc2","Type":"ContainerDied","Data":"675b897f9420a967dc80308977aed9333b25599f5d107852cb9bd5ddda74b80b"} Oct 30 22:29:14.496928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.496911 4305 scope.go:122] "RemoveContainer" containerID="99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1" Oct 30 22:29:14.497227 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:14.497179 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1\": container with ID starting with 99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1 not found: ID does not exist" containerID="99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1" Oct 30 22:29:14.497302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.497215 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1"} err="failed to get container status \"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1\": rpc error: code = NotFound desc = could not find container \"99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1\": container with ID starting with 99244de260f2e4dc08e8775e4824145fe9b730f03fd22be6411ecbd1066c82e1 not found: ID does not exist" Oct 30 22:29:14.497302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.497239 4305 scope.go:122] "RemoveContainer" containerID="f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae" Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569357 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569438 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569491 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569638 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569700 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.569827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569769 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569872 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569917 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.569961 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570061 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570180 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570215 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.570265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570257 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.570635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570287 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570314 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570381 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\") pod \"9f7d9091-a50e-4d27-af57-e098d735a24e\" (UID: \"9f7d9091-a50e-4d27-af57-e098d735a24e\") " Oct 30 22:29:14.570635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.570433 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\") pod \"c4ea0083-1fd5-4023-8c74-9d2978145014\" (UID: \"c4ea0083-1fd5-4023-8c74-9d2978145014\") " Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571190 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571220 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571235 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571251 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571265 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571279 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571292 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135ec54c-ac83-45ca-ae90-8f1e90855cc2-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.573822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.571307 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxd4s\" (UniqueName: \"kubernetes.io/projected/135ec54c-ac83-45ca-ae90-8f1e90855cc2-kube-api-access-pxd4s\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.595829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.593960 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.605380 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.605341 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "kube-api-access-dl8k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.612485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.612451 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.619818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.612948 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.614047 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.624953 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.626897 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.632044 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.632498 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.633176 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:14.660077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.633455 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688473 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688507 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688524 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688540 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8k7\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-kube-api-access-dl8k7\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688555 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7d9091-a50e-4d27-af57-e098d735a24e-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688568 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688581 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688594 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688607 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688622 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ea0083-1fd5-4023-8c74-9d2978145014-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.693834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.688636 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.728858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.714601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.728858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.724022 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:14.731133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.730968 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.749774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.749017 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.787306 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.787454 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd" pod "9f7d9091-a50e-4d27-af57-e098d735a24e" (UID: "9f7d9091-a50e-4d27-af57-e098d735a24e"). InnerVolumeSpecName "kube-api-access-bmhwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794880 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794910 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794926 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794942 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794958 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmhwd\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-kube-api-access-bmhwd\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.795079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.794971 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7d9091-a50e-4d27-af57-e098d735a24e-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.898165 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.887927 4305 scope.go:122] "RemoveContainer" containerID="f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae" Oct 30 22:29:14.898165 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.889140 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy" pod "c4ea0083-1fd5-4023-8c74-9d2978145014" (UID: "c4ea0083-1fd5-4023-8c74-9d2978145014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:14.946516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.932228 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ea0083-1fd5-4023-8c74-9d2978145014-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:14.946516 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:14.932406 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae\": container with ID starting with f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae not found: ID does not exist" containerID="f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae" Oct 30 22:29:14.946516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.932445 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae"} err="failed to get container status \"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae\": rpc error: code = NotFound desc = could not find container \"f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae\": container with ID starting with f3953ca48478405c59f060cbc980efc6df0b69e085332ec24d6bff0babf55fae not found: ID does not exist" Oct 30 22:29:14.946516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:14.932471 4305 scope.go:122] "RemoveContainer" containerID="e87e41739019f271082b8cff313e41466ed29841bc5f7d7c64f6e0342bbd47b6" Oct 30 22:29:15.049714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.049691 4305 scope.go:122] "RemoveContainer" containerID="735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f" Oct 30 22:29:15.069860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.069833 4305 scope.go:122] "RemoveContainer" containerID="735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f" Oct 30 22:29:15.070307 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:15.070177 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f\": container with ID starting with 735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f not found: ID does not exist" containerID="735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f" Oct 30 22:29:15.070307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.070210 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f"} err="failed to get container status \"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f\": rpc error: code = NotFound desc = could not find container \"735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f\": container with ID starting with 735c446fe5225cca9bf097b6fb3f6b9404f1641903f14540de22f9acce4ab94f not found: ID does not exist" Oct 30 22:29:15.070307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.070229 4305 scope.go:122] "RemoveContainer" containerID="efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54" Oct 30 22:29:15.094933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.088917 4305 scope.go:122] "RemoveContainer" containerID="efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54" Oct 30 22:29:15.097846 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:15.095980 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54\": container with ID starting with efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54 not found: ID does not exist" containerID="efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54" Oct 30 22:29:15.097846 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.096021 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54"} err="failed to get container status \"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54\": rpc error: code = NotFound desc = could not find container \"efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54\": container with ID starting with efb106f9877aa96f26a07665f6fee6154f2c9633b3a9115a6bf98560ff75bf54 not found: ID does not exist" Oct 30 22:29:15.097846 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.096046 4305 scope.go:122] "RemoveContainer" containerID="46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d" Oct 30 22:29:15.119114 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.118709 4305 scope.go:122] "RemoveContainer" containerID="46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d" Oct 30 22:29:15.119724 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:15.119610 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d\": container with ID starting with 46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d not found: ID does not exist" containerID="46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d" Oct 30 22:29:15.119724 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.119649 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d"} err="failed to get container status \"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d\": rpc error: code = NotFound desc = could not find container \"46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d\": container with ID starting with 46cc3aac021693684fd4b943d700eb2760f743f4091d70460bf76649bec57c9d not found: ID does not exist" Oct 30 22:29:15.287084 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.287048 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:29:15.361834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.361687 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-rzws5"] Oct 30 22:29:15.409135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.409096 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:29:15.413714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.413687 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-gxq6c"] Oct 30 22:29:15.519125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.519086 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109521bf-2196-4621-a52c-f0330c45f94d" path="/var/lib/kubelet/pods/109521bf-2196-4621-a52c-f0330c45f94d/volumes" Oct 30 22:29:15.519680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.519656 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" path="/var/lib/kubelet/pods/48e539ea-bd5d-4ab6-9b2f-8d93851fec5b/volumes" Oct 30 22:29:15.520246 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.520225 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" path="/var/lib/kubelet/pods/7f20bbd8-30b8-419c-8534-4c83e76665be/volumes" Oct 30 22:29:15.521088 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.521067 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" path="/var/lib/kubelet/pods/a9b78c95-1ae0-4e46-83f7-7217547a25ed/volumes" Oct 30 22:29:15.521580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.521559 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" path="/var/lib/kubelet/pods/e65fccc6-fd3f-4757-b5bb-bd2254834e73/volumes" Oct 30 22:29:15.528919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.528897 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:29:15.532341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.532318 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-d2rbz"] Oct 30 22:29:15.646594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.646517 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:29:15.650789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.650762 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-rm7tj"] Oct 30 22:29:15.792830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.781500 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:29:15.801647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.801615 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mngqp"] Oct 30 22:29:15.917477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.917396 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:29:15.923613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:15.923587 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-p4sbs"] Oct 30 22:29:16.112027 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:16.111991 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:29:16.355251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:16.355215 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:16.368051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:16.368021 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:16.533237 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:16.533208 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:29:17.522714 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:17.522680 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" path="/var/lib/kubelet/pods/135ec54c-ac83-45ca-ae90-8f1e90855cc2/volumes" Oct 30 22:29:17.523243 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:17.523228 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91068286-7de3-4d59-8d59-20180fe271c0" path="/var/lib/kubelet/pods/91068286-7de3-4d59-8d59-20180fe271c0/volumes" Oct 30 22:29:17.523678 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:17.523664 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" path="/var/lib/kubelet/pods/9f7d9091-a50e-4d27-af57-e098d735a24e/volumes" Oct 30 22:29:17.524338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:17.524310 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" path="/var/lib/kubelet/pods/c4ea0083-1fd5-4023-8c74-9d2978145014/volumes" Oct 30 22:29:20.454394 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.454363 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:29:20.454978 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.454965 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" Oct 30 22:29:20.455030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.454979 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" Oct 30 22:29:20.455030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455001 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" Oct 30 22:29:20.455030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455006 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" Oct 30 22:29:20.455030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455018 4305 cpu_manager.go:414] "containerMap: removing container" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" Oct 30 22:29:20.455030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455024 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455039 4305 cpu_manager.go:414] "containerMap: removing container" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455044 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455058 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455063 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455073 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455078 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455086 4305 cpu_manager.go:414] "containerMap: removing container" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455091 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455096 4305 cpu_manager.go:414] "containerMap: removing container" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455101 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455109 4305 cpu_manager.go:414] "containerMap: removing container" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" Oct 30 22:29:20.455187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455113 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455195 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7d9091-a50e-4d27-af57-e098d735a24e" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455207 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="109521bf-2196-4621-a52c-f0330c45f94d" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455216 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b78c95-1ae0-4e46-83f7-7217547a25ed" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455223 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65fccc6-fd3f-4757-b5bb-bd2254834e73" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455229 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e539ea-bd5d-4ab6-9b2f-8d93851fec5b" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455234 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="135ec54c-ac83-45ca-ae90-8f1e90855cc2" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455240 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="91068286-7de3-4d59-8d59-20180fe271c0" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455248 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f20bbd8-30b8-419c-8534-4c83e76665be" containerName="istio-proxy" Oct 30 22:29:20.455531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.455256 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ea0083-1fd5-4023-8c74-9d2978145014" containerName="istio-proxy" Oct 30 22:29:20.461684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.461665 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.516724 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.516688 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:29:20.562990 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.562953 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.563154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.563068 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.563154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.563101 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5c2f68-serving-cert\"" Oct 30 22:29:20.563260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.563225 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.563313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.563267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hp6t\" (UniqueName: \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.665075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.665036 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.665241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.665145 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.665241 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:20.665195 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-serving-cert: secret "maas-api-e2e-shared-b-w2-5c2f68-serving-cert" not found Oct 30 22:29:20.665312 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:20.665268 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls podName:6f5d36d0-8ca4-4e96-b3c3-6b9648996932 nodeName:}" failed. No retries permitted until 2026-10-30 22:29:21.165248064 +0000 UTC m=+1681.760336351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls") pod "maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" (UID: "6f5d36d0-8ca4-4e96-b3c3-6b9648996932") : secret "maas-api-e2e-shared-b-w2-5c2f68-serving-cert" not found Oct 30 22:29:20.665367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.665330 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.665399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.665382 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hp6t\" (UniqueName: \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.665916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.665892 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.667547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.667522 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:20.834418 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:20.834303 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hp6t\" (UniqueName: \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:21.187977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:21.187873 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:21.190386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:21.190355 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:21.376552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:21.376521 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:21.762076 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:29:21.762042 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5d36d0_8ca4_4e96_b3c3_6b9648996932.slice/crio-3197a4c65e7c318a9bfaaa0f13bc5b7e4d3ab3d3a83bff8ddf69664cb2f0a2ee WatchSource:0}: Error finding container 3197a4c65e7c318a9bfaaa0f13bc5b7e4d3ab3d3a83bff8ddf69664cb2f0a2ee: Status 404 returned error can't find the container with id 3197a4c65e7c318a9bfaaa0f13bc5b7e4d3ab3d3a83bff8ddf69664cb2f0a2ee Oct 30 22:29:21.762626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:21.762599 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:29:21.763747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:21.763724 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:29:22.561313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:22.561275 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" event={"ID":"6f5d36d0-8ca4-4e96-b3c3-6b9648996932","Type":"ContainerStarted","Data":"3197a4c65e7c318a9bfaaa0f13bc5b7e4d3ab3d3a83bff8ddf69664cb2f0a2ee"} Oct 30 22:29:23.638945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:23.635390 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" event={"ID":"6f5d36d0-8ca4-4e96-b3c3-6b9648996932","Type":"ContainerStarted","Data":"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620"} Oct 30 22:29:23.638945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:23.636540 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:23.761338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:23.761282 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" podStartSLOduration=2.764374847 podStartE2EDuration="3.761268861s" totalImagesPullingTime="996.894014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:23.758701247 +0000 UTC m=+1684.353789552" watchObservedRunningTime="2026-10-30 22:29:23.761268861 +0000 UTC m=+1684.356357168" Oct 30 22:29:30.700244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:30.700211 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:29:46.110268 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:46.110223 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:29:47.960202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:47.960170 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:29:47.964609 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:47.964585 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.021099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.021045 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:29:48.054861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.054826 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.055029 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.054966 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.055103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.055036 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfnb\" (UniqueName: \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.055161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.055116 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.069039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.069015 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-bbd10f-serving-cert\"" Oct 30 22:29:48.156580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.156549 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.156722 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:48.156694 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-serving-cert: secret "maas-api-e2e-shared-b-w5-bbd10f-serving-cert" not found Oct 30 22:29:48.156722 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.156708 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.156810 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:29:48.156760 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls podName:8046dff0-a710-4847-88df-0b4ece37203c nodeName:}" failed. No retries permitted until 2026-10-30 22:29:48.656742702 +0000 UTC m=+1709.251830989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls") pod "maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" (UID: "8046dff0-a710-4847-88df-0b4ece37203c") : secret "maas-api-e2e-shared-b-w5-bbd10f-serving-cert" not found Oct 30 22:29:48.156870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.156789 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfnb\" (UniqueName: \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.156928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.156897 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.157448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.157425 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.159219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.159201 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.235453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.235324 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfnb\" (UniqueName: \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.676999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.676913 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.679417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.679387 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:48.878577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:48.878544 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:49.259254 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:29:49.259211 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8046dff0_a710_4847_88df_0b4ece37203c.slice/crio-ad238846ef7f4090f88383c048dc55af8457d2ecb576d62cc2cf7476f30f3830 WatchSource:0}: Error finding container ad238846ef7f4090f88383c048dc55af8457d2ecb576d62cc2cf7476f30f3830: Status 404 returned error can't find the container with id ad238846ef7f4090f88383c048dc55af8457d2ecb576d62cc2cf7476f30f3830 Oct 30 22:29:49.259770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:49.259744 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:29:49.926409 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:49.926372 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" event={"ID":"8046dff0-a710-4847-88df-0b4ece37203c","Type":"ContainerStarted","Data":"ad238846ef7f4090f88383c048dc55af8457d2ecb576d62cc2cf7476f30f3830"} Oct 30 22:29:50.971737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:50.971385 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" event={"ID":"8046dff0-a710-4847-88df-0b4ece37203c","Type":"ContainerStarted","Data":"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027"} Oct 30 22:29:50.972390 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:50.972364 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:29:51.096870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:51.096793 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" podStartSLOduration=3.219986428 podStartE2EDuration="4.096778052s" totalImagesPullingTime="876.791624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:51.093023891 +0000 UTC m=+1711.688112196" watchObservedRunningTime="2026-10-30 22:29:51.096778052 +0000 UTC m=+1711.691866358" Oct 30 22:29:57.994726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:29:57.994697 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:30:01.806387 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.806351 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:30:01.812058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.812034 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:01.866309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.866278 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:30:01.910308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.910275 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:01.910470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.910332 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:01.910470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.910388 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzkz\" (UniqueName: \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:01.910580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.910561 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:01.914229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:01.914208 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-71c235-serving-cert\"" Oct 30 22:30:02.012486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.012448 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.012663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.012625 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.012706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.012679 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.012706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.012696 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzkz\" (UniqueName: \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.012829 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:02.012790 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-serving-cert: secret "maas-api-e2e-shared-a-w0-71c235-serving-cert" not found Oct 30 22:30:02.012932 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:02.012882 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls podName:e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd nodeName:}" failed. No retries permitted until 2026-10-30 22:30:02.512862191 +0000 UTC m=+1723.107950492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls") pod "maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" (UID: "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd") : secret "maas-api-e2e-shared-a-w0-71c235-serving-cert" not found Oct 30 22:30:02.013458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.013432 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.014983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.014962 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.082550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.082435 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzkz\" (UniqueName: \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.532430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.532396 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.534832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.534788 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:02.726436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:02.726403 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:03.118228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:03.118183 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:30:03.157717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:03.157680 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" event={"ID":"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd","Type":"ContainerStarted","Data":"4c2fa22361a8e3baac3f6847a81656d73aaf4563d98af034b6f647de8cdebe69"} Oct 30 22:30:04.165866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:04.165829 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" event={"ID":"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd","Type":"ContainerStarted","Data":"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b"} Oct 30 22:30:04.166251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:04.165951 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:04.330477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:04.330373 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" podStartSLOduration=2.440767723 podStartE2EDuration="3.330359161s" totalImagesPullingTime="889.591438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.326854927 +0000 UTC m=+1724.921943230" watchObservedRunningTime="2026-10-30 22:30:04.330359161 +0000 UTC m=+1724.925447466" Oct 30 22:30:10.178271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:10.178240 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:30:11.004653 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.004621 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:30:11.008973 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.008948 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.065085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.065052 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:30:11.102397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.102366 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-73735c-serving-cert\"" Oct 30 22:30:11.176648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.176614 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.176853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.176827 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qg6\" (UniqueName: \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.176906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.176896 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.176975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.176958 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.288031 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27qg6\" (UniqueName: \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.288143 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.288185 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.288287 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.289273 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:11.289692 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-serving-cert: secret "maas-api-e2e-shared-a-w3-73735c-serving-cert" not found Oct 30 22:30:11.291045 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:11.289814 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls podName:e4867f6d-f77d-437f-9df5-eac91d93b67b nodeName:}" failed. No retries permitted until 2026-10-30 22:30:11.789777288 +0000 UTC m=+1732.384865588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" (UID: "e4867f6d-f77d-437f-9df5-eac91d93b67b") : secret "maas-api-e2e-shared-a-w3-73735c-serving-cert" not found Oct 30 22:30:11.341516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.341480 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.363715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.363644 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qg6\" (UniqueName: \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.797203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.797165 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.799612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.799585 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:11.923241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:11.923199 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:12.306160 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:30:12.306115 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4867f6d_f77d_437f_9df5_eac91d93b67b.slice/crio-1581c7a94dd23a2b3243605c0e964a0872e452b51e86356b53f151613426f36d WatchSource:0}: Error finding container 1581c7a94dd23a2b3243605c0e964a0872e452b51e86356b53f151613426f36d: Status 404 returned error can't find the container with id 1581c7a94dd23a2b3243605c0e964a0872e452b51e86356b53f151613426f36d Oct 30 22:30:12.307320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:12.307291 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:30:13.227821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:13.227780 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" event={"ID":"e4867f6d-f77d-437f-9df5-eac91d93b67b","Type":"ContainerStarted","Data":"1581c7a94dd23a2b3243605c0e964a0872e452b51e86356b53f151613426f36d"} Oct 30 22:30:14.235482 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:14.235437 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" event={"ID":"e4867f6d-f77d-437f-9df5-eac91d93b67b","Type":"ContainerStarted","Data":"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1"} Oct 30 22:30:14.235971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:14.235505 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:14.360114 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:14.360052 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" podStartSLOduration=3.465400119 podStartE2EDuration="4.360035311s" totalImagesPullingTime="894.635192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:14.356775775 +0000 UTC m=+1734.951864080" watchObservedRunningTime="2026-10-30 22:30:14.360035311 +0000 UTC m=+1734.955123619" Oct 30 22:30:16.123882 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:16.123845 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:30:20.248947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.248878 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:30:20.358588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.358554 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:30:20.363073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.363053 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.412678 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.412639 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:30:20.417536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.417494 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a3600a-serving-cert\"" Oct 30 22:30:20.434824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.434573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnb6s\" (UniqueName: \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.434824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.434773 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.435124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.434872 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.435124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.434987 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.537791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.537691 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.537791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.537786 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.538042 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:20.537897 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-serving-cert: secret "maas-api-e2e-api-a-a3600a-serving-cert" not found Oct 30 22:30:20.538042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.537910 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.538042 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:20.537976 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls podName:a096c4fd-0a25-4091-a461-6c03c58e6d1c nodeName:}" failed. No retries permitted until 2026-10-30 22:30:21.037960168 +0000 UTC m=+1741.633048453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls") pod "maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" (UID: "a096c4fd-0a25-4091-a461-6c03c58e6d1c") : secret "maas-api-e2e-api-a-a3600a-serving-cert" not found Oct 30 22:30:20.538211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.538052 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnb6s\" (UniqueName: \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.538564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.538547 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.540524 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.540501 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:20.621885 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:20.615306 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnb6s\" (UniqueName: \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:21.053469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:21.053427 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:21.056114 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:21.056085 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") pod \"maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:21.280563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:21.280531 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:21.670129 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:30:21.670096 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda096c4fd_0a25_4091_a461_6c03c58e6d1c.slice/crio-b8d940485a6e24f8bd91a13022b50ad22882e219e55352ffd3950951a344f47c WatchSource:0}: Error finding container b8d940485a6e24f8bd91a13022b50ad22882e219e55352ffd3950951a344f47c: Status 404 returned error can't find the container with id b8d940485a6e24f8bd91a13022b50ad22882e219e55352ffd3950951a344f47c Oct 30 22:30:21.671706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:21.671681 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:30:22.293557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:22.293514 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" event={"ID":"a096c4fd-0a25-4091-a461-6c03c58e6d1c","Type":"ContainerStarted","Data":"b8d940485a6e24f8bd91a13022b50ad22882e219e55352ffd3950951a344f47c"} Oct 30 22:30:23.450685 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:23.450115 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" event={"ID":"a096c4fd-0a25-4091-a461-6c03c58e6d1c","Type":"ContainerStarted","Data":"b8b61df8059bddab30b3d3aef96d72af9822f35fc34c0e44fe5dca915b6838ae"} Oct 30 22:30:23.451252 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:23.450623 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:23.574052 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:23.573991 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" podStartSLOduration=2.673198595 podStartE2EDuration="3.573978027s" totalImagesPullingTime="900.779432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:23.573091084 +0000 UTC m=+1744.168179391" watchObservedRunningTime="2026-10-30 22:30:23.573978027 +0000 UTC m=+1744.169066333" Oct 30 22:30:29.463061 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:30:29.463030 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:30:46.112361 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:30:46.112310 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:31:16.110289 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:31:16.110250 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:31:19.767959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:19.767916 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:31:19.769160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:19.769136 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:31:20.144842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:20.144743 4305 scope.go:122] "RemoveContainer" containerID="c12dd4afb9cc20174f8e33ba64c1b1f854c1c13ad39990233d497c871037678a" Oct 30 22:31:31.425718 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:31.425684 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:31:31.426205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:31.425906 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" containerID="cri-o://c40bf2f1ac9c5a3582cbba4e2df27bfe9f6917586176373e9f7e597088c6a4d6" gracePeriod=30 Oct 30 22:31:31.477542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:31.477509 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:31:31.477723 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:31.477702 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" containerID="cri-o://26315c6cdf9788f1e14cdf3fd72fc06184e03608d44ce9197ed3332172c9b6c7" gracePeriod=30 Oct 30 22:31:36.079076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.079040 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:31:36.085072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.085049 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113161 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113196 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113350 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113454 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113514 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113583 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113545 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113672 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fc4\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113717 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.113767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.113735 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.146118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.146081 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:31:36.189437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.189402 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-29bbe735-openshift-default-dockercfg-sgjmc\"" Oct 30 22:31:36.214914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.214877 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.214914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.214914 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215011 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215028 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215101 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215145 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215449 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215215 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215449 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215241 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.215449 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215303 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64fc4\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.216076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215637 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.216076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.215661 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.216076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.216034 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.216630 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.216544 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.216728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.216649 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.218184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.218159 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.218477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.218453 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.278365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.278328 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.278521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.278504 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fc4\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.409779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.409687 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:36.612277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.612244 4305 generic.go:356] "Generic (PLEG): container finished" podID="8da63901-13b2-489c-82de-42e638560747" containerID="c40bf2f1ac9c5a3582cbba4e2df27bfe9f6917586176373e9f7e597088c6a4d6" exitCode=0 Oct 30 22:31:36.612414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.612336 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" event={"ID":"8da63901-13b2-489c-82de-42e638560747","Type":"ContainerDied","Data":"c40bf2f1ac9c5a3582cbba4e2df27bfe9f6917586176373e9f7e597088c6a4d6"} Oct 30 22:31:36.616655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.616021 4305 generic.go:356] "Generic (PLEG): container finished" podID="0829a02d-a461-46cc-9452-8216602a604b" containerID="26315c6cdf9788f1e14cdf3fd72fc06184e03608d44ce9197ed3332172c9b6c7" exitCode=0 Oct 30 22:31:36.616655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.616064 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" event={"ID":"0829a02d-a461-46cc-9452-8216602a604b","Type":"ContainerDied","Data":"26315c6cdf9788f1e14cdf3fd72fc06184e03608d44ce9197ed3332172c9b6c7"} Oct 30 22:31:36.791012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.790986 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:31:36.814314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.814287 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:31:36.837564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837539 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.837693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837578 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.837693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837611 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.837693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837641 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.837693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837678 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837779 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837821 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837862 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837892 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.837925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837886 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.838221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.837975 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.838221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838014 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.838221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838075 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.838221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838133 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.838221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838194 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:36.838461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838226 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.838461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838306 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\") pod \"8da63901-13b2-489c-82de-42e638560747\" (UID: \"8da63901-13b2-489c-82de-42e638560747\") " Oct 30 22:31:36.838461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838336 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.838461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838398 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.838651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838526 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\") pod \"0829a02d-a461-46cc-9452-8216602a604b\" (UID: \"0829a02d-a461-46cc-9452-8216602a604b\") " Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.838854 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839073 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839495 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da63901-13b2-489c-82de-42e638560747-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839549 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839565 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839581 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839596 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839681 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839703 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.839944 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.840155 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.840560 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:36.840915 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.840888 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.840991 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.840968 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.841072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.840988 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.841072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.841031 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:36.841400 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.841381 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "kube-api-access-hlxnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.841470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.841417 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token" pod "8da63901-13b2-489c-82de-42e638560747" (UID: "8da63901-13b2-489c-82de-42e638560747"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.842523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.842499 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:36.842674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.842659 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98" pod "0829a02d-a461-46cc-9452-8216602a604b" (UID: "0829a02d-a461-46cc-9452-8216602a604b"). InnerVolumeSpecName "kube-api-access-rgr98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.941610 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941574 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0829a02d-a461-46cc-9452-8216602a604b-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941616 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941632 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941647 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgr98\" (UniqueName: \"kubernetes.io/projected/0829a02d-a461-46cc-9452-8216602a604b-kube-api-access-rgr98\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941664 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941677 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941691 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941705 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da63901-13b2-489c-82de-42e638560747-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941719 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0829a02d-a461-46cc-9452-8216602a604b-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941733 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxnf\" (UniqueName: \"kubernetes.io/projected/8da63901-13b2-489c-82de-42e638560747-kube-api-access-hlxnf\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941746 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941759 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da63901-13b2-489c-82de-42e638560747-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.941773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.941773 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0829a02d-a461-46cc-9452-8216602a604b-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.970626 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:36.970588 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435c8a54_6eb5_4035_99b0_c9e640467500.slice/crio-4b01128b0e6e0b003d6b76ddf25e5ecc1ea62d6aa6b148c087f58b3b36976d26 WatchSource:0}: Error finding container 4b01128b0e6e0b003d6b76ddf25e5ecc1ea62d6aa6b148c087f58b3b36976d26: Status 404 returned error can't find the container with id 4b01128b0e6e0b003d6b76ddf25e5ecc1ea62d6aa6b148c087f58b3b36976d26 Oct 30 22:31:36.973835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.973788 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:31:36.974284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.974260 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:36.974326 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.974317 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:36.974362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:36.974347 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:37.624559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.624522 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" event={"ID":"435c8a54-6eb5-4035-99b0-c9e640467500","Type":"ContainerStarted","Data":"ffe34c0180e11bbd225ce590a0e31f6df8237b612d42214147e843548da7e0fc"} Oct 30 22:31:37.625056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.624586 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" event={"ID":"435c8a54-6eb5-4035-99b0-c9e640467500","Type":"ContainerStarted","Data":"4b01128b0e6e0b003d6b76ddf25e5ecc1ea62d6aa6b148c087f58b3b36976d26"} Oct 30 22:31:37.626016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.625994 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" Oct 30 22:31:37.626016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.626005 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd" event={"ID":"8da63901-13b2-489c-82de-42e638560747","Type":"ContainerDied","Data":"29053804aa8e2a974e9e384bc9013f9f5af21c80d6a339a4e94d5da68ef297d5"} Oct 30 22:31:37.626196 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.626054 4305 scope.go:122] "RemoveContainer" containerID="c40bf2f1ac9c5a3582cbba4e2df27bfe9f6917586176373e9f7e597088c6a4d6" Oct 30 22:31:37.627537 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.627508 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" Oct 30 22:31:37.627667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.627550 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v" event={"ID":"0829a02d-a461-46cc-9452-8216602a604b","Type":"ContainerDied","Data":"cbb919c8b091516606c44ab5d5ace00534dd66d2b766046f08b4f4ca571d6902"} Oct 30 22:31:37.636449 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.636427 4305 scope.go:122] "RemoveContainer" containerID="26315c6cdf9788f1e14cdf3fd72fc06184e03608d44ce9197ed3332172c9b6c7" Oct 30 22:31:37.750651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.750589 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podStartSLOduration=1.750575414 podStartE2EDuration="1.750575414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:37.749688406 +0000 UTC m=+1818.344776715" watchObservedRunningTime="2026-10-30 22:31:37.750575414 +0000 UTC m=+1818.345663720" Oct 30 22:31:37.876619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.876527 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:31:37.880529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.880503 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bdh5v"] Oct 30 22:31:37.994587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.994557 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:31:37.999410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:37.999388 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-v7krd"] Oct 30 22:31:38.475820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.470095 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:38.529815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.527996 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:38.529815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.528085 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:38.532100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.532072 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:31:38.532534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.532504 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" containerID="cri-o://4ae9bd24ab65f0359bb37ab8a8ecc04cead47d388deaa5f7b19f10722c182761" gracePeriod=30 Oct 30 22:31:38.826908 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.826498 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:31:38.826908 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:38.826759 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" containerID="cri-o://d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e" gracePeriod=30 Oct 30 22:31:39.411667 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.411630 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:39.411933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.411701 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:39.417851 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.417821 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:31:39.418057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.418034 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" containerID="cri-o://443808b7fb8ee8a315c33fe22cda5c104f46551fe963e5ae6ed93f3f8a13fc87" gracePeriod=30 Oct 30 22:31:39.520138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.520101 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0829a02d-a461-46cc-9452-8216602a604b" path="/var/lib/kubelet/pods/0829a02d-a461-46cc-9452-8216602a604b/volumes" Oct 30 22:31:39.520691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.520674 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da63901-13b2-489c-82de-42e638560747" path="/var/lib/kubelet/pods/8da63901-13b2-489c-82de-42e638560747/volumes" Oct 30 22:31:39.775929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.775876 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:31:39.776877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:39.776771 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" containerID="cri-o://6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120" gracePeriod=30 Oct 30 22:31:40.411075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:40.411043 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:40.411482 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:40.411111 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:41.410715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:41.410680 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:41.410902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:41.410731 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:42.043058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:42.043020 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:42.043443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:42.043079 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:42.411501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:42.411411 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:42.411501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:42.411474 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:43.411458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.411416 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:31:43.411965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.411475 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:31:43.703687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.703660 4305 generic.go:356] "Generic (PLEG): container finished" podID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerID="4ae9bd24ab65f0359bb37ab8a8ecc04cead47d388deaa5f7b19f10722c182761" exitCode=0 Oct 30 22:31:43.703864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.703735 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" event={"ID":"002d6c06-0d4a-46d2-b140-c7ad80daa567","Type":"ContainerDied","Data":"4ae9bd24ab65f0359bb37ab8a8ecc04cead47d388deaa5f7b19f10722c182761"} Oct 30 22:31:43.928519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.928490 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:31:43.950535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950504 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.950659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950599 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.950723 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950675 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.950862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.950962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950889 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.951026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.950973 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:43.951026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951006 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.951141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.951141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951111 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.951252 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951156 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\") pod \"002d6c06-0d4a-46d2-b140-c7ad80daa567\" (UID: \"002d6c06-0d4a-46d2-b140-c7ad80daa567\") " Oct 30 22:31:43.951584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951555 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.951873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951781 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.951985 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.951901 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.952338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952313 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.952635 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952611 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.952720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952641 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.952720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952657 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/002d6c06-0d4a-46d2-b140-c7ad80daa567-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.952720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952670 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.952720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.952683 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.954186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.954159 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:43.955350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.955316 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:43.958270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.958245 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "kube-api-access-6sb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:43.958870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:43.958844 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo" pod "002d6c06-0d4a-46d2-b140-c7ad80daa567" (UID: "002d6c06-0d4a-46d2-b140-c7ad80daa567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.049971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.049915 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:44.050136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.049997 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:44.055087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.055053 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.055208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.055093 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.055208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.055108 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sb6n\" (UniqueName: \"kubernetes.io/projected/002d6c06-0d4a-46d2-b140-c7ad80daa567-kube-api-access-6sb6n\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.055208 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.055123 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/002d6c06-0d4a-46d2-b140-c7ad80daa567-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.107902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.107872 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:31:44.156528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.156490 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.156696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.156558 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.156696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.156632 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.156701 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157010 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.156946 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157187 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157821 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157786 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157888 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.158156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.157931 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\") pod \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\" (UID: \"2be5ad7d-52cb-4a46-8d9c-51f4334a577f\") " Oct 30 22:31:44.159298 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.158664 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.159298 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.158694 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.159298 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.158936 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.159298 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.159026 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.159298 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.159246 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.160044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.160019 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "kube-api-access-qhl98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.160256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.160225 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.161164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.161138 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.161452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.161427 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy" pod "2be5ad7d-52cb-4a46-8d9c-51f4334a577f" (UID: "2be5ad7d-52cb-4a46-8d9c-51f4334a577f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.259857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259776 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.259857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259824 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhl98\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-kube-api-access-qhl98\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.259857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259842 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.259857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259852 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.259857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259861 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.260115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259870 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.260115 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.259878 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2be5ad7d-52cb-4a46-8d9c-51f4334a577f-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.491843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.489131 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:44.504163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.496399 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:44.533410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.533238 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:31:44.797720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.796324 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" event={"ID":"002d6c06-0d4a-46d2-b140-c7ad80daa567","Type":"ContainerDied","Data":"78c7cbbe2412b0408770d457960f4e502b880bc32f4c0f350f9fdbbf5dbc5920"} Oct 30 22:31:44.797720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.796389 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr" Oct 30 22:31:44.797720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.796392 4305 scope.go:122] "RemoveContainer" containerID="4ae9bd24ab65f0359bb37ab8a8ecc04cead47d388deaa5f7b19f10722c182761" Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.808907 4305 generic.go:356] "Generic (PLEG): container finished" podID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerID="443808b7fb8ee8a315c33fe22cda5c104f46551fe963e5ae6ed93f3f8a13fc87" exitCode=0 Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.809005 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" event={"ID":"4bb77e60-ea37-4985-8d4b-70ee10be88f8","Type":"ContainerDied","Data":"443808b7fb8ee8a315c33fe22cda5c104f46551fe963e5ae6ed93f3f8a13fc87"} Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.810730 4305 generic.go:356] "Generic (PLEG): container finished" podID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerID="d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e" exitCode=0 Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.810821 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.812737 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" event={"ID":"2be5ad7d-52cb-4a46-8d9c-51f4334a577f","Type":"ContainerDied","Data":"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e"} Oct 30 22:31:44.813863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.812783 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws" event={"ID":"2be5ad7d-52cb-4a46-8d9c-51f4334a577f","Type":"ContainerDied","Data":"873db526451a87e239847ea4ba68e8ff65f8fd769fa7f7a6be8d9814455aecdc"} Oct 30 22:31:44.838512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.838487 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:31:44.839610 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.839562 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" Oct 30 22:31:44.839752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.839738 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" Oct 30 22:31:44.839908 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.839896 4305 cpu_manager.go:414] "containerMap: removing container" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" Oct 30 22:31:44.839996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.839985 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" Oct 30 22:31:44.840081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840072 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" Oct 30 22:31:44.840153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840144 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" Oct 30 22:31:44.840235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840226 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" Oct 30 22:31:44.840306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840296 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" Oct 30 22:31:44.840547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840534 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" containerName="istio-proxy" Oct 30 22:31:44.840640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840631 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da63901-13b2-489c-82de-42e638560747" containerName="istio-proxy" Oct 30 22:31:44.840729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840720 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0829a02d-a461-46cc-9452-8216602a604b" containerName="istio-proxy" Oct 30 22:31:44.840827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.840817 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" containerName="istio-proxy" Oct 30 22:31:44.845095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.845076 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:44.922193 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.901365 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6116bd-dockercfg-44tnn\"" Oct 30 22:31:44.922193 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.906058 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6116bd\"" Oct 30 22:31:44.959417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.959369 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9dl\" (UniqueName: \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:44.959658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:44.959542 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.103820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.099035 4305 scope.go:122] "RemoveContainer" containerID="d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e" Oct 30 22:31:45.114975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.106191 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:31:45.152582 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.150214 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.152582 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.150632 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9dl\" (UniqueName: \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.154844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.153293 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.192582 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.192556 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:31:45.208789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.208760 4305 scope.go:122] "RemoveContainer" containerID="d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e" Oct 30 22:31:45.216380 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:31:45.216291 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e\": container with ID starting with d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e not found: ID does not exist" containerID="d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e" Oct 30 22:31:45.216380 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.216346 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e"} err="failed to get container status \"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e\": rpc error: code = NotFound desc = could not find container \"d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e\": container with ID starting with d16a929a390449d220a8d5f0f47396ffb621b52df7259b026b643d79b384518e not found: ID does not exist" Oct 30 22:31:45.217605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.217535 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9dl\" (UniqueName: \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\") pod \"payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.258998 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259111 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259144 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259170 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259204 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259294 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259362 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259415 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.264822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.259495 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\") pod \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\" (UID: \"4bb77e60-ea37-4985-8d4b-70ee10be88f8\") " Oct 30 22:31:45.286871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.274024 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:45.286871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.274572 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.286871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.275112 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.287758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.287214 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.297853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.291315 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.297853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.291376 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:31:45.297853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.297173 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.298119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.297895 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-9bkws"] Oct 30 22:31:45.298242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.298220 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:45.298701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.298678 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "kube-api-access-pqxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:45.299957 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.299932 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token" pod "4bb77e60-ea37-4985-8d4b-70ee10be88f8" (UID: "4bb77e60-ea37-4985-8d4b-70ee10be88f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.371934 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.371983 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372006 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372029 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372042 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372064 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372077 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372091 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb77e60-ea37-4985-8d4b-70ee10be88f8-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.373829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.372105 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxfl\" (UniqueName: \"kubernetes.io/projected/4bb77e60-ea37-4985-8d4b-70ee10be88f8-kube-api-access-pqxfl\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.514473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.514211 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:45.519887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.515097 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:31:45.519887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.515150 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mrtlr"] Oct 30 22:31:45.543171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.543138 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002d6c06-0d4a-46d2-b140-c7ad80daa567" path="/var/lib/kubelet/pods/002d6c06-0d4a-46d2-b140-c7ad80daa567/volumes" Oct 30 22:31:45.544099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.544057 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be5ad7d-52cb-4a46-8d9c-51f4334a577f" path="/var/lib/kubelet/pods/2be5ad7d-52cb-4a46-8d9c-51f4334a577f/volumes" Oct 30 22:31:45.563838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.563786 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:31:45.718872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.718837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.718891 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.718929 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.718970 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719010 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719068 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719164 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719202 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\") pod \"427438a0-38f5-431c-a210-25230914a082\" (UID: \"427438a0-38f5-431c-a210-25230914a082\") " Oct 30 22:31:45.719482 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719120 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.719682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719651 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.719682 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719671 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.719898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.719995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719936 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.719995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719957 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.719995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.719973 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.720158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.720142 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:45.721551 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.721530 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:45.721663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.721580 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "kube-api-access-7zk9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:45.721663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.721651 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:45.721779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.721694 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy" pod "427438a0-38f5-431c-a210-25230914a082" (UID: "427438a0-38f5-431c-a210-25230914a082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:45.820750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.820708 4305 generic.go:356] "Generic (PLEG): container finished" podID="427438a0-38f5-431c-a210-25230914a082" containerID="6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120" exitCode=0 Oct 30 22:31:45.820962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.820759 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" Oct 30 22:31:45.820962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.820793 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" event={"ID":"427438a0-38f5-431c-a210-25230914a082","Type":"ContainerDied","Data":"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120"} Oct 30 22:31:45.820962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.820876 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv" event={"ID":"427438a0-38f5-431c-a210-25230914a082","Type":"ContainerDied","Data":"94c3ebc1e8a7e7d8ce545d30905881fe838801657c9282b700f62ba512a3f260"} Oct 30 22:31:45.820962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.820912 4305 scope.go:122] "RemoveContainer" containerID="6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120" Oct 30 22:31:45.821177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821154 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zk9z\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-kube-api-access-7zk9z\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.821271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821182 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.821271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821198 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/427438a0-38f5-431c-a210-25230914a082-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.821271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821213 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/427438a0-38f5-431c-a210-25230914a082-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.821271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821227 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/427438a0-38f5-431c-a210-25230914a082-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.821271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.821240 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/427438a0-38f5-431c-a210-25230914a082-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.824906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.824872 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" event={"ID":"4bb77e60-ea37-4985-8d4b-70ee10be88f8","Type":"ContainerDied","Data":"8fb501732b72b9ee661bc15ba5e797770fade9766e6f7c05484f88c5a65ef56e"} Oct 30 22:31:45.825015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.824882 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf" Oct 30 22:31:45.833691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.833675 4305 scope.go:122] "RemoveContainer" containerID="6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120" Oct 30 22:31:45.833992 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:31:45.833970 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120\": container with ID starting with 6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120 not found: ID does not exist" containerID="6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120" Oct 30 22:31:45.833992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.833996 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120"} err="failed to get container status \"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120\": rpc error: code = NotFound desc = could not find container \"6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120\": container with ID starting with 6e490ea61625f171c0734ff20894a7c0e5f082bba179bb13b41e284b42867120 not found: ID does not exist" Oct 30 22:31:45.834177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.834013 4305 scope.go:122] "RemoveContainer" containerID="443808b7fb8ee8a315c33fe22cda5c104f46551fe963e5ae6ed93f3f8a13fc87" Oct 30 22:31:45.922358 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:45.922313 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b86cdbb_380a_45dd_8abf_9a7e4bd429b9.slice/crio-94ed4ab08215723575cb39120cc769a4c777eb4cf2f26cc81c913cf0724f24f5 WatchSource:0}: Error finding container 94ed4ab08215723575cb39120cc769a4c777eb4cf2f26cc81c913cf0724f24f5: Status 404 returned error can't find the container with id 94ed4ab08215723575cb39120cc769a4c777eb4cf2f26cc81c913cf0724f24f5 Oct 30 22:31:45.922647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.922626 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:31:45.997469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:45.997441 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:31:46.005000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.004974 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-9b6qv"] Oct 30 22:31:46.077611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.077585 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:31:46.078232 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078209 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" Oct 30 22:31:46.078279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078233 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" Oct 30 22:31:46.078279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078255 4305 cpu_manager.go:414] "containerMap: removing container" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" Oct 30 22:31:46.078279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078261 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" Oct 30 22:31:46.078372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078358 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="427438a0-38f5-431c-a210-25230914a082" containerName="istio-proxy" Oct 30 22:31:46.078402 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.078374 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" containerName="istio-proxy" Oct 30 22:31:46.083269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.083252 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.116393 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:31:46.116360 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:31:46.123471 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.123447 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:31:46.125857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.125830 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj9pg\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.125961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.125878 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126004 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.125972 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126013 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126130 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126116 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126151 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126253 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126338 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126308 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.126393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.126368 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.127303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.127283 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tp9kf"] Oct 30 22:31:46.139540 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.139519 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d57980-openshift-default-dockercfg-k2d54\"" Oct 30 22:31:46.142784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.142618 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:31:46.294226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294181 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294287 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294403 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294477 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294441 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294504 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294586 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj9pg\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294627 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294760 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.294891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.294868 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.295434 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.295405 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.295918 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.295890 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.296176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.296146 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.296400 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.296379 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.296637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.296614 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.298463 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.298176 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.298463 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.298400 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.357923 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.357861 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.358018 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.357943 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj9pg\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\") pod \"e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.414320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.414299 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:46.836423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.836385 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" event={"ID":"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9","Type":"ContainerStarted","Data":"94ed4ab08215723575cb39120cc769a4c777eb4cf2f26cc81c913cf0724f24f5"} Oct 30 22:31:46.936397 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:46.936366 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b667a3_896d_4485_a6aa_2cdc954aeb3f.slice/crio-d06c49637703c0dfbd0f90de28ce4357e745393e37921d34812701db48d833fc WatchSource:0}: Error finding container d06c49637703c0dfbd0f90de28ce4357e745393e37921d34812701db48d833fc: Status 404 returned error can't find the container with id d06c49637703c0dfbd0f90de28ce4357e745393e37921d34812701db48d833fc Oct 30 22:31:46.937297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.937272 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:31:46.939251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.939213 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:46.939358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.939292 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:46.939358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:46.939328 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:47.523064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:47.523029 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427438a0-38f5-431c-a210-25230914a082" path="/var/lib/kubelet/pods/427438a0-38f5-431c-a210-25230914a082/volumes" Oct 30 22:31:47.523738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:47.523722 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb77e60-ea37-4985-8d4b-70ee10be88f8" path="/var/lib/kubelet/pods/4bb77e60-ea37-4985-8d4b-70ee10be88f8/volumes" Oct 30 22:31:47.847733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:47.847646 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerStarted","Data":"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637"} Oct 30 22:31:47.848172 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:47.847739 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerStarted","Data":"d06c49637703c0dfbd0f90de28ce4357e745393e37921d34812701db48d833fc"} Oct 30 22:31:48.047098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:48.047046 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podStartSLOduration=2.047033158 podStartE2EDuration="2.047033158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:48.023303636 +0000 UTC m=+1828.618391943" watchObservedRunningTime="2026-10-30 22:31:48.047033158 +0000 UTC m=+1828.642121463" Oct 30 22:31:48.416818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:48.415897 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:48.449833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:48.436717 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:48.855839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:48.855812 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:48.857154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:48.857126 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:31:49.863335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:49.863292 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" event={"ID":"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9","Type":"ContainerStarted","Data":"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19"} Oct 30 22:31:49.863845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:49.863377 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:31:49.986615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:49.986569 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" podStartSLOduration=2.7400254569999998 podStartE2EDuration="5.986553208s" totalImagesPullingTime="3.246527751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:49.984834593 +0000 UTC m=+1830.579922900" watchObservedRunningTime="2026-10-30 22:31:49.986553208 +0000 UTC m=+1830.581641517" Oct 30 22:31:52.052483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.052447 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:31:52.056899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.056878 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.112665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.112634 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b5313f-openshift-default-dockercfg-5v6zx\"" Oct 30 22:31:52.119440 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119406 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119541 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119672 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119636 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6w4\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119782 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119877 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119922 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.119986 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.119950 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.120036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.120023 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.120075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.120059 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.167566 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.167537 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235234 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235375 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235433 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6w4\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235514 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235598 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235642 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235721 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.235764 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.242818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.236358 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.272539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.272503 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.273420 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.273389 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.279579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.273888 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.282598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.282382 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.289872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.289830 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.299842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.294333 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.335927 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.335854 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6w4\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.342917 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.342892 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\") pod \"e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.370955 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.370916 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:52.868838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.868792 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:31:52.901867 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:52.901832 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27da875_b9a4_43b0_b969_4a439c63f31d.slice/crio-82b5a7279b83edc554599efd03f52039916f22f746f51978fa88906e85d06bd5 WatchSource:0}: Error finding container 82b5a7279b83edc554599efd03f52039916f22f746f51978fa88906e85d06bd5: Status 404 returned error can't find the container with id 82b5a7279b83edc554599efd03f52039916f22f746f51978fa88906e85d06bd5 Oct 30 22:31:52.904592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.904562 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:52.904690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.904630 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:52.904690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:52.904660 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:31:53.897930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:53.897883 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerStarted","Data":"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35"} Oct 30 22:31:53.898384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:53.897977 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerStarted","Data":"82b5a7279b83edc554599efd03f52039916f22f746f51978fa88906e85d06bd5"} Oct 30 22:31:54.111078 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.111026 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podStartSLOduration=2.111011695 podStartE2EDuration="2.111011695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:54.109459416 +0000 UTC m=+1834.704547723" watchObservedRunningTime="2026-10-30 22:31:54.111011695 +0000 UTC m=+1834.706100000" Oct 30 22:31:54.376561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.376526 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:54.382094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.382070 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:54.905030 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.904995 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:54.905851 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.905831 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:31:54.970993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:54.970952 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:31:55.043057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.043025 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:31:55.043552 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.043523 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.134106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.127839 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d57980-dockercfg-6dt4p\"" Oct 30 22:31:55.134106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.130147 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:31:55.135028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.126443 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d57980\"" Oct 30 22:31:55.136404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.136384 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.217081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.213699 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l4j2\" (UniqueName: \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.217081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.215725 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.224830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.221938 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:31:55.317010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.316972 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l4j2\" (UniqueName: \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.317175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.317041 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88sh2\" (UniqueName: \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.317175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.317101 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.317284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.317264 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.317857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.317839 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.380934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.380903 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l4j2\" (UniqueName: \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\") pod \"payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.418621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.418589 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.418777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.418734 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88sh2\" (UniqueName: \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.419295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.419272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.425241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.425216 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:55.484213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.483967 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88sh2\" (UniqueName: \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\") pod \"payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.508494 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.494209 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:55.807978 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:55.807951 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8192c484_3f3f_4025_aa8c_93498cb87f93.slice/crio-b78f270e4b9643f116fbe23f3337a5dc3f7962dc4b46b7a298e94cd459d490bb WatchSource:0}: Error finding container b78f270e4b9643f116fbe23f3337a5dc3f7962dc4b46b7a298e94cd459d490bb: Status 404 returned error can't find the container with id b78f270e4b9643f116fbe23f3337a5dc3f7962dc4b46b7a298e94cd459d490bb Oct 30 22:31:55.809053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.809021 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:31:55.909163 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:55.909134 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23999a27_4ae7_4fc9_878d_4cccce9a4632.slice/crio-95671965c002fa59d886c6c76ab8d8a1ee32a4831ac53deea7ab961480e0eab6 WatchSource:0}: Error finding container 95671965c002fa59d886c6c76ab8d8a1ee32a4831ac53deea7ab961480e0eab6: Status 404 returned error can't find the container with id 95671965c002fa59d886c6c76ab8d8a1ee32a4831ac53deea7ab961480e0eab6 Oct 30 22:31:55.911520 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.911494 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:31:55.914501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.914471 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" event={"ID":"8192c484-3f3f-4025-aa8c-93498cb87f93","Type":"ContainerStarted","Data":"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c"} Oct 30 22:31:55.918637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.914844 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" event={"ID":"8192c484-3f3f-4025-aa8c-93498cb87f93","Type":"ContainerStarted","Data":"b78f270e4b9643f116fbe23f3337a5dc3f7962dc4b46b7a298e94cd459d490bb"} Oct 30 22:31:55.918637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:55.915405 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:31:56.039060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:56.039009 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" podStartSLOduration=2.038995809 podStartE2EDuration="2.038995809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:56.03644758 +0000 UTC m=+1836.631535885" watchObservedRunningTime="2026-10-30 22:31:56.038995809 +0000 UTC m=+1836.634084115" Oct 30 22:31:56.923485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:56.923446 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" event={"ID":"23999a27-4ae7-4fc9-878d-4cccce9a4632","Type":"ContainerStarted","Data":"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987"} Oct 30 22:31:56.923919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:56.923504 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" event={"ID":"23999a27-4ae7-4fc9-878d-4cccce9a4632","Type":"ContainerStarted","Data":"95671965c002fa59d886c6c76ab8d8a1ee32a4831ac53deea7ab961480e0eab6"} Oct 30 22:31:57.050621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:57.050556 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" podStartSLOduration=2.050536345 podStartE2EDuration="2.050536345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:57.044910609 +0000 UTC m=+1837.639998927" watchObservedRunningTime="2026-10-30 22:31:57.050536345 +0000 UTC m=+1837.645624651" Oct 30 22:31:57.930542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:57.930512 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:31:58.722283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.722253 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-d47ec9"] Oct 30 22:31:58.726734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.726709 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:31:58.768972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.768934 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcrz9\" (UniqueName: \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\") pod \"test-curl-7135-d47ec9\" (UID: \"c6291e24-a0e8-4996-9aa5-7eb47731dc06\") " pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:31:58.866820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.866242 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-d47ec9"] Oct 30 22:31:58.866820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.866475 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9v4z\"" Oct 30 22:31:58.871010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.870964 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcrz9\" (UniqueName: \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\") pod \"test-curl-7135-d47ec9\" (UID: \"c6291e24-a0e8-4996-9aa5-7eb47731dc06\") " pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:31:58.934330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:58.934297 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcrz9\" (UniqueName: \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\") pod \"test-curl-7135-d47ec9\" (UID: \"c6291e24-a0e8-4996-9aa5-7eb47731dc06\") " pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:31:59.058874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:59.046349 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:31:59.606378 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:31:59.606344 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6291e24_a0e8_4996_9aa5_7eb47731dc06.slice/crio-67463fbfa8a04e860c5c5cfd56a50be5a142c1174dfb6925a4cd5d08ca97bc8d WatchSource:0}: Error finding container 67463fbfa8a04e860c5c5cfd56a50be5a142c1174dfb6925a4cd5d08ca97bc8d: Status 404 returned error can't find the container with id 67463fbfa8a04e860c5c5cfd56a50be5a142c1174dfb6925a4cd5d08ca97bc8d Oct 30 22:31:59.639584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:59.639544 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-d47ec9"] Oct 30 22:31:59.955217 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:31:59.955169 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-d47ec9" event={"ID":"c6291e24-a0e8-4996-9aa5-7eb47731dc06","Type":"ContainerStarted","Data":"67463fbfa8a04e860c5c5cfd56a50be5a142c1174dfb6925a4cd5d08ca97bc8d"} Oct 30 22:32:00.608236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.608202 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:32:00.613782 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.613761 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.738821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.734616 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b5313f\"" Oct 30 22:32:00.738821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.735718 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.738821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.735888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjmf\" (UniqueName: \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.738821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.736206 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b5313f-dockercfg-w2hgs\"" Oct 30 22:32:00.878145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.840782 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.878145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.873815 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjmf\" (UniqueName: \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.909257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.909221 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:32:00.916243 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.915040 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:32:00.926645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.926622 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:32:00.933147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.933094 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:32:00.933307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.933295 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:00.934109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.934068 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:00.944627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:00.944595 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjmf\" (UniqueName: \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\") pod \"payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:01.077329 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.077291 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvz8n\" (UniqueName: \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.077752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.077428 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.179012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.178928 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.179254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.179222 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvz8n\" (UniqueName: \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.179894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.179868 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.229790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.229759 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:01.244501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.244471 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvz8n\" (UniqueName: \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\") pod \"payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.276778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.276744 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.624705 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:32:01.624672 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caadcd7_56b0_4e12_b49e_3cf9e7f679d4.slice/crio-5de9cf60a855b66d27d7e3d9cc71d529196baab3c021659896e495065e00738e WatchSource:0}: Error finding container 5de9cf60a855b66d27d7e3d9cc71d529196baab3c021659896e495065e00738e: Status 404 returned error can't find the container with id 5de9cf60a855b66d27d7e3d9cc71d529196baab3c021659896e495065e00738e Oct 30 22:32:01.626946 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.626918 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:32:01.680579 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:32:01.680551 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc311579e_2b1a_45c7_aa09_e6e0a6652d47.slice/crio-2e6341b4ec64009a460d9d2393d6ff8786b08002b627c96bdcb80f546647a101 WatchSource:0}: Error finding container 2e6341b4ec64009a460d9d2393d6ff8786b08002b627c96bdcb80f546647a101: Status 404 returned error can't find the container with id 2e6341b4ec64009a460d9d2393d6ff8786b08002b627c96bdcb80f546647a101 Oct 30 22:32:01.681939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.681913 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:32:01.973777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.973742 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" event={"ID":"c311579e-2b1a-45c7-aa09-e6e0a6652d47","Type":"ContainerStarted","Data":"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271"} Oct 30 22:32:01.974154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.973848 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" event={"ID":"c311579e-2b1a-45c7-aa09-e6e0a6652d47","Type":"ContainerStarted","Data":"2e6341b4ec64009a460d9d2393d6ff8786b08002b627c96bdcb80f546647a101"} Oct 30 22:32:01.974154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.974009 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:01.976219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.976189 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" event={"ID":"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4","Type":"ContainerStarted","Data":"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1"} Oct 30 22:32:01.976362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.976240 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" event={"ID":"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4","Type":"ContainerStarted","Data":"5de9cf60a855b66d27d7e3d9cc71d529196baab3c021659896e495065e00738e"} Oct 30 22:32:01.976362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:01.976270 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:02.097521 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:02.097387 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" podStartSLOduration=2.097374499 podStartE2EDuration="2.097374499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:02.093774282 +0000 UTC m=+1842.688862589" watchObservedRunningTime="2026-10-30 22:32:02.097374499 +0000 UTC m=+1842.692462804" Oct 30 22:32:02.250259 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:02.250159 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" podStartSLOduration=2.250142198 podStartE2EDuration="2.250142198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:02.229206558 +0000 UTC m=+1842.824294867" watchObservedRunningTime="2026-10-30 22:32:02.250142198 +0000 UTC m=+1842.845230504" Oct 30 22:32:02.984313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:02.984282 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-d47ec9" event={"ID":"c6291e24-a0e8-4996-9aa5-7eb47731dc06","Type":"ContainerStarted","Data":"19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96"} Oct 30 22:32:03.104017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:03.103965 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-d47ec9" podStartSLOduration=1.841015703 podStartE2EDuration="5.103950843s" totalImagesPullingTime="3.26293514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:03.103193099 +0000 UTC m=+1843.698281407" watchObservedRunningTime="2026-10-30 22:32:03.103950843 +0000 UTC m=+1843.699039149" Oct 30 22:32:04.594195 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:04.594162 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-d47ec9"] Oct 30 22:32:04.599898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:04.599863 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-d47ec9"] Oct 30 22:32:04.997570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:04.997531 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-d47ec9" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" containerName="test-curl-7135-d47ec9" containerID="cri-o://19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96" gracePeriod=2 Oct 30 22:32:05.050626 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:05.050586 4305 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-d47ec9\" is forbidden: User \"system:node:ip-10-0-134-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-125.us-west-2.compute.internal' and this object" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:32:06.926836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:06.926783 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:32:07.268957 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:07.268933 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:32:07.391260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:07.391225 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\" (UniqueName: \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\") pod \"c6291e24-a0e8-4996-9aa5-7eb47731dc06\" (UID: \"c6291e24-a0e8-4996-9aa5-7eb47731dc06\") " Oct 30 22:32:07.393399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:07.393367 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9" pod "c6291e24-a0e8-4996-9aa5-7eb47731dc06" (UID: "c6291e24-a0e8-4996-9aa5-7eb47731dc06"). InnerVolumeSpecName "kube-api-access-qcrz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:07.493471 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:07.493395 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcrz9\" (UniqueName: \"kubernetes.io/projected/c6291e24-a0e8-4996-9aa5-7eb47731dc06-kube-api-access-qcrz9\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:07.520106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:07.520073 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" path="/var/lib/kubelet/pods/c6291e24-a0e8-4996-9aa5-7eb47731dc06/volumes" Oct 30 22:32:08.020160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.020119 4305 generic.go:356] "Generic (PLEG): container finished" podID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" containerID="19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96" exitCode=137 Oct 30 22:32:08.020642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.020183 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-d47ec9" Oct 30 22:32:08.020642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.020204 4305 scope.go:122] "RemoveContainer" containerID="19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96" Oct 30 22:32:08.031230 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.031210 4305 scope.go:122] "RemoveContainer" containerID="19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96" Oct 30 22:32:08.031533 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:08.031510 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96\": container with ID starting with 19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96 not found: ID does not exist" containerID="19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96" Oct 30 22:32:08.031617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.031544 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96"} err="failed to get container status \"19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96\": rpc error: code = NotFound desc = could not find container \"19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96\": container with ID starting with 19f708e2386db63fa26c0069b3fc3d960dfec3ba8ceb0936e276bb2a4d7d0a96 not found: ID does not exist" Oct 30 22:32:08.941817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:08.941759 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:32:12.987235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:12.987199 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:32:12.987698 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:12.987266 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:32:16.110278 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:16.110241 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:32:28.452180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.452148 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:32:28.452868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.452852 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" containerName="test-curl-7135-d47ec9" Oct 30 22:32:28.452920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.452869 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" containerName="test-curl-7135-d47ec9" Oct 30 22:32:28.453017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.453007 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6291e24-a0e8-4996-9aa5-7eb47731dc06" containerName="test-curl-7135-d47ec9" Oct 30 22:32:28.456558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.456542 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.513683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.513649 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:32:28.560827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.560766 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-721c75-serving-cert\"" Oct 30 22:32:28.590566 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.590515 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.590744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.590572 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.590834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.590764 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.590977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.590953 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtbw\" (UniqueName: \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.693013 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.693080 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.693208 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.693361 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtbw\" (UniqueName: \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.694588 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:28.694698 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-serving-cert: secret "maas-api-e2e-api-a-721c75-serving-cert" not found Oct 30 22:32:28.707625 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:28.694774 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls podName:d190b2f2-f6a2-405d-b1a7-1d39f4fc635c nodeName:}" failed. No retries permitted until 2026-10-30 22:32:29.194754657 +0000 UTC m=+1869.789842968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls") pod "maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" (UID: "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c") : secret "maas-api-e2e-api-a-721c75-serving-cert" not found Oct 30 22:32:28.764816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.761362 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:28.769820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:28.766372 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtbw\" (UniqueName: \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:29.206983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:29.206951 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:29.209572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:29.209540 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") pod \"maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:29.371484 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:29.371441 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:29.751974 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:32:29.751942 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd190b2f2_f6a2_405d_b1a7_1d39f4fc635c.slice/crio-dd9a15ab5a048d85cbf006c4411a010aa16cfb4d2cd3a6110865076ca3d0111c WatchSource:0}: Error finding container dd9a15ab5a048d85cbf006c4411a010aa16cfb4d2cd3a6110865076ca3d0111c: Status 404 returned error can't find the container with id dd9a15ab5a048d85cbf006c4411a010aa16cfb4d2cd3a6110865076ca3d0111c Oct 30 22:32:29.754315 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:29.754273 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:32:30.176149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:30.176059 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" event={"ID":"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c","Type":"ContainerStarted","Data":"dd9a15ab5a048d85cbf006c4411a010aa16cfb4d2cd3a6110865076ca3d0111c"} Oct 30 22:32:31.185747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:31.185689 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" event={"ID":"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c","Type":"ContainerStarted","Data":"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193"} Oct 30 22:32:31.186687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:31.185827 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:31.311064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:31.311009 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" podStartSLOduration=2.4682025 podStartE2EDuration="3.310995215s" totalImagesPullingTime="842.792715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:31.307121898 +0000 UTC m=+1871.902210205" watchObservedRunningTime="2026-10-30 22:32:31.310995215 +0000 UTC m=+1871.906083521" Oct 30 22:32:37.204225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:37.204185 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:32:46.112613 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:32:46.112575 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:32:47.022715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.022683 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:32:47.027341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.027318 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064370 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064431 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064615 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064662 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064831 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064896 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.064955 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.064943 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.065039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.065025 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.065078 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.065057 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45bf\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.140669 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.140635 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:32:47.166560 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166513 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.166774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166690 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.166774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166732 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.166977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166860 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.166977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166892 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.166977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.166926 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167144 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167009 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167144 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167050 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45bf\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167147 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167222 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167260 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167234 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167618 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.167869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.167736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.168074 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.168052 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.169783 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.169760 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.170166 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.170144 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.230348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.230310 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.230779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.230759 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45bf\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zz7mk\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.342575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.342488 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:47.778291 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:32:47.778259 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb31495d_361f_46f5_a08a_c7b0b08fc24d.slice/crio-60c7717c78f9b1788bd44fdb53ddc63a7d45c754e62db0d1dc7123a17f9d58e8 WatchSource:0}: Error finding container 60c7717c78f9b1788bd44fdb53ddc63a7d45c754e62db0d1dc7123a17f9d58e8: Status 404 returned error can't find the container with id 60c7717c78f9b1788bd44fdb53ddc63a7d45c754e62db0d1dc7123a17f9d58e8 Oct 30 22:32:47.780184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.780158 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:32:47.781566 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.781529 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:32:47.781668 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.781635 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:32:47.781732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:47.781682 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:32:48.328838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.327573 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" event={"ID":"db31495d-361f-46f5-a08a-c7b0b08fc24d","Type":"ContainerStarted","Data":"8cb0fd44a3baf98049a6e706f89070aa8e983df94ebacd298678c9b3bf1fc168"} Oct 30 22:32:48.328838 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.327647 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" event={"ID":"db31495d-361f-46f5-a08a-c7b0b08fc24d","Type":"ContainerStarted","Data":"60c7717c78f9b1788bd44fdb53ddc63a7d45c754e62db0d1dc7123a17f9d58e8"} Oct 30 22:32:48.347747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.342876 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:48.347747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.344769 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:48.347747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.344837 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:48.474888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:48.474764 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podStartSLOduration=2.474742564 podStartE2EDuration="2.474742564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:48.470344419 +0000 UTC m=+1889.065432736" watchObservedRunningTime="2026-10-30 22:32:48.474742564 +0000 UTC m=+1889.069830873" Oct 30 22:32:49.343617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:49.343585 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:49.344048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:49.343647 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:50.343909 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:50.343875 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:50.344318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:50.343924 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:51.342971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:51.342933 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:51.343149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:51.343009 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:52.343385 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:52.343340 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:52.343912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:52.343422 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:53.342855 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:53.342820 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:53.343051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:53.342886 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:54.346207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:54.346163 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:54.346902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:54.346862 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:55.343402 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:55.343369 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:32:55.343572 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:55.343431 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:32:56.563544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:56.563508 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:56.565058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:56.565038 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:32:56.565671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:32:56.565645 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:33:00.872963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.872929 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:33:00.877167 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.877149 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:00.933162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.933131 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:33:00.968647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.968608 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:00.968883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.968811 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:00.969022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.968999 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:00.969119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.969100 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dw4\" (UniqueName: \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:00.979393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:00.979367 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6116bd-serving-cert\"" Oct 30 22:33:01.070889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.070856 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.071056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.070935 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.071056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.071036 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.071142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.071085 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dw4\" (UniqueName: \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.071201 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:01.071183 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-serving-cert: secret "maas-api-e2e-shared-b-w0-6116bd-serving-cert" not found Oct 30 22:33:01.071278 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:01.071266 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls podName:ea471d82-1606-4c1f-964f-b41b0dbcfda5 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:01.571246621 +0000 UTC m=+1902.166334905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" (UID: "ea471d82-1606-4c1f-964f-b41b0dbcfda5") : secret "maas-api-e2e-shared-b-w0-6116bd-serving-cert" not found Oct 30 22:33:01.071619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.071596 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.073558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.073541 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.161871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.158895 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dw4\" (UniqueName: \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.589725 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.589689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.592117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.592093 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:01.792621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:01.792582 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:02.178487 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:33:02.178456 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea471d82_1606_4c1f_964f_b41b0dbcfda5.slice/crio-8210e69bd71a845d5dab83cd9698ef57ab69d345b2db7efdcc3e38bc2c621a0e WatchSource:0}: Error finding container 8210e69bd71a845d5dab83cd9698ef57ab69d345b2db7efdcc3e38bc2c621a0e: Status 404 returned error can't find the container with id 8210e69bd71a845d5dab83cd9698ef57ab69d345b2db7efdcc3e38bc2c621a0e Oct 30 22:33:02.179113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:02.179085 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:33:02.445268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:02.445234 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" event={"ID":"ea471d82-1606-4c1f-964f-b41b0dbcfda5","Type":"ContainerStarted","Data":"8210e69bd71a845d5dab83cd9698ef57ab69d345b2db7efdcc3e38bc2c621a0e"} Oct 30 22:33:03.453628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:03.453591 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" event={"ID":"ea471d82-1606-4c1f-964f-b41b0dbcfda5","Type":"ContainerStarted","Data":"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc"} Oct 30 22:33:03.454142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:03.453846 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:03.580938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:03.580884 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" podStartSLOduration=2.7117194380000003 podStartE2EDuration="3.580869669s" totalImagesPullingTime="869.150231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:03.576423602 +0000 UTC m=+1904.171511899" watchObservedRunningTime="2026-10-30 22:33:03.580869669 +0000 UTC m=+1904.175957974" Oct 30 22:33:09.466436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:09.466404 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:33:16.174986 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:16.174949 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:33:29.021874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.021835 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:33:29.026396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.026376 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.083568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.083532 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:33:29.087577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.087544 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.087791 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.087753 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.087943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.087882 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.088016 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.087998 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvpx\" (UniqueName: \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.128721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.128687 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b5313f-serving-cert\"" Oct 30 22:33:29.189676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.189636 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvpx\" (UniqueName: \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.189913 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.189859 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.190017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.189997 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.190098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.190052 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.190157 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:29.190088 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-serving-cert: secret "maas-api-e2e-api-b-b5313f-serving-cert" not found Oct 30 22:33:29.190203 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:29.190160 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls podName:07847b61-406f-4ab5-9888-3d2ad97d1cc1 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:29.690138703 +0000 UTC m=+1930.285226994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls") pod "maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" (UID: "07847b61-406f-4ab5-9888-3d2ad97d1cc1") : secret "maas-api-e2e-api-b-b5313f-serving-cert" not found Oct 30 22:33:29.190733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.190699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.192461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.192439 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.308869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.307883 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvpx\" (UniqueName: \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.723354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.723323 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.725943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.725914 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") pod \"maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:29.941849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:29.941819 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:30.327352 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:33:30.327323 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07847b61_406f_4ab5_9888_3d2ad97d1cc1.slice/crio-03011dfdd3b01c53b37e4782326035fffb2aeff7a6c35473212002f4c374624b WatchSource:0}: Error finding container 03011dfdd3b01c53b37e4782326035fffb2aeff7a6c35473212002f4c374624b: Status 404 returned error can't find the container with id 03011dfdd3b01c53b37e4782326035fffb2aeff7a6c35473212002f4c374624b Oct 30 22:33:30.328608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:30.328582 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:33:30.719741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:30.719709 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" event={"ID":"07847b61-406f-4ab5-9888-3d2ad97d1cc1","Type":"ContainerStarted","Data":"03011dfdd3b01c53b37e4782326035fffb2aeff7a6c35473212002f4c374624b"} Oct 30 22:33:31.728408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:31.728371 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" event={"ID":"07847b61-406f-4ab5-9888-3d2ad97d1cc1","Type":"ContainerStarted","Data":"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa"} Oct 30 22:33:31.728856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:31.728467 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:31.872932 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:31.872684 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" podStartSLOduration=3.045662348 podStartE2EDuration="3.872662696s" totalImagesPullingTime="827.000348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:31.84923917 +0000 UTC m=+1932.444327489" watchObservedRunningTime="2026-10-30 22:33:31.872662696 +0000 UTC m=+1932.467751003" Oct 30 22:33:37.742226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:33:37.742194 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:33:46.110982 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:33:46.110946 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:34:14.121837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.121698 4305 generic.go:356] "Generic (PLEG): container finished" podID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerID="c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35" exitCode=0 Oct 30 22:34:14.121837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.121754 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerDied","Data":"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35"} Oct 30 22:34:14.122405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.122315 4305 scope.go:122] "RemoveContainer" containerID="c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35" Oct 30 22:34:14.124184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.124154 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:14.124290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.124225 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:14.124290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:14.124260 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:15.133173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:15.133133 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerStarted","Data":"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae"} Oct 30 22:34:15.376658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:15.376616 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:34:15.377973 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:15.376818 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:15.377973 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:15.376881 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:16.112449 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:34:16.112408 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:34:16.141185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.141149 4305 generic.go:356] "Generic (PLEG): container finished" podID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerID="d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637" exitCode=0 Oct 30 22:34:16.141654 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.141227 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerDied","Data":"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637"} Oct 30 22:34:16.141856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.141836 4305 scope.go:122] "RemoveContainer" containerID="d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637" Oct 30 22:34:16.143432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.143404 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:16.143511 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.143471 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:16.143511 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.143505 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:34:16.383205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.382339 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:16.383205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:16.382390 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:17.188280 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.188238 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerStarted","Data":"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265"} Oct 30 22:34:17.377679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.377198 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:17.377679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.377264 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:17.415334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.415300 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:34:17.417122 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.417095 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:17.417252 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:17.417146 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:18.380818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:18.380168 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:18.380818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:18.380227 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:18.414941 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:18.414906 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:18.415126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:18.414959 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:18.862089 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:18.862054 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:34:19.378824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:19.376944 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:19.378824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:19.377019 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:19.415082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:19.415047 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:19.415576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:19.415542 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:20.380824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:20.380192 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:34:20.380824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:20.380279 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 22:34:20.419254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:20.419216 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:20.419861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:20.419826 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:21.426821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:21.419182 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:21.426821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:21.419243 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:21.584853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:21.582106 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:34:21.584853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:21.584108 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:34:21.598828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:21.585554 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:34:22.415157 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:22.415120 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:34:22.415336 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:22.415191 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:34:23.420105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:23.420074 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:34:23.421275 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:23.421251 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:34:36.596306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:36.596269 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:34:36.596840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:36.596506 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" podUID="adefb7d0-117f-40a1-957b-e7b109de05d6" containerName="maas-api" containerID="cri-o://afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201" gracePeriod=30 Oct 30 22:34:37.060820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.060624 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:34:37.166850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.166811 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\") pod \"adefb7d0-117f-40a1-957b-e7b109de05d6\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " Oct 30 22:34:37.167041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.166990 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\" (UniqueName: \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\") pod \"adefb7d0-117f-40a1-957b-e7b109de05d6\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " Oct 30 22:34:37.167102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.167057 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\") pod \"adefb7d0-117f-40a1-957b-e7b109de05d6\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " Oct 30 22:34:37.167223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.167206 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") pod \"adefb7d0-117f-40a1-957b-e7b109de05d6\" (UID: \"adefb7d0-117f-40a1-957b-e7b109de05d6\") " Oct 30 22:34:37.193853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.191918 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls" pod "adefb7d0-117f-40a1-957b-e7b109de05d6" (UID: "adefb7d0-117f-40a1-957b-e7b109de05d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:37.193853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.192364 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca" pod "adefb7d0-117f-40a1-957b-e7b109de05d6" (UID: "adefb7d0-117f-40a1-957b-e7b109de05d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:37.234945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.227525 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs" pod "adefb7d0-117f-40a1-957b-e7b109de05d6" (UID: "adefb7d0-117f-40a1-957b-e7b109de05d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:37.269563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.269381 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v" pod "adefb7d0-117f-40a1-957b-e7b109de05d6" (UID: "adefb7d0-117f-40a1-957b-e7b109de05d6"). InnerVolumeSpecName "kube-api-access-zqf7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:37.271354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.271322 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:37.271485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.271359 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adefb7d0-117f-40a1-957b-e7b109de05d6-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:37.271485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.271375 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf7v\" (UniqueName: \"kubernetes.io/projected/adefb7d0-117f-40a1-957b-e7b109de05d6-kube-api-access-zqf7v\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:37.271485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.271389 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adefb7d0-117f-40a1-957b-e7b109de05d6-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:37.426287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.426257 4305 generic.go:356] "Generic (PLEG): container finished" podID="adefb7d0-117f-40a1-957b-e7b109de05d6" containerID="afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201" exitCode=0 Oct 30 22:34:37.426475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.426305 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" Oct 30 22:34:37.426475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.426345 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" event={"ID":"adefb7d0-117f-40a1-957b-e7b109de05d6","Type":"ContainerDied","Data":"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201"} Oct 30 22:34:37.426475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.426414 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n" event={"ID":"adefb7d0-117f-40a1-957b-e7b109de05d6","Type":"ContainerDied","Data":"8f56989a21b28fd8352ac349ffb15a432a11d1efae40ddf77a02e61672078f71"} Oct 30 22:34:37.426475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.426443 4305 scope.go:122] "RemoveContainer" containerID="afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201" Oct 30 22:34:37.437649 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.437630 4305 scope.go:122] "RemoveContainer" containerID="afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201" Oct 30 22:34:37.437971 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:34:37.437951 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201\": container with ID starting with afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201 not found: ID does not exist" containerID="afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201" Oct 30 22:34:37.438093 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.437983 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201"} err="failed to get container status \"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201\": rpc error: code = NotFound desc = could not find container \"afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201\": container with ID starting with afc99d091dc15f58bdbba0f4510b7d413d1de6dec3c5b24fbfc178159aab9201 not found: ID does not exist" Oct 30 22:34:37.653881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.649854 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:34:37.658818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:37.655408 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-vnl6n"] Oct 30 22:34:39.520759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:34:39.520724 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adefb7d0-117f-40a1-957b-e7b109de05d6" path="/var/lib/kubelet/pods/adefb7d0-117f-40a1-957b-e7b109de05d6/volumes" Oct 30 22:34:46.111865 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:34:46.111829 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:35:16.111193 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:35:16.111151 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:35:43.001954 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.001919 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:35:43.004688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.003962 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" podUID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" containerName="maas-api" containerID="cri-o://6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193" gracePeriod=30 Oct 30 22:35:43.516900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.502766 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:35:43.631156 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.631121 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\" (UniqueName: \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\") pod \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " Oct 30 22:35:43.631326 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.631176 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\") pod \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " Oct 30 22:35:43.631392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.631375 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") pod \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " Oct 30 22:35:43.631456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.631442 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\") pod \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\" (UID: \"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c\") " Oct 30 22:35:43.632075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.632045 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca" pod "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" (UID: "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:43.634396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.634371 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw" pod "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" (UID: "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c"). InnerVolumeSpecName "kube-api-access-2qtbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:43.634576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.634422 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls" pod "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" (UID: "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:43.636226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.636199 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs" pod "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" (UID: "d190b2f2-f6a2-405d-b1a7-1d39f4fc635c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:43.732963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.732938 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.732963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.732963 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtbw\" (UniqueName: \"kubernetes.io/projected/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-kube-api-access-2qtbw\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.733117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.732973 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.733117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.732982 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:43.995830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.994641 4305 generic.go:356] "Generic (PLEG): container finished" podID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" containerID="6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193" exitCode=0 Oct 30 22:35:43.995830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.994917 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" event={"ID":"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c","Type":"ContainerDied","Data":"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193"} Oct 30 22:35:43.995830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.994971 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" event={"ID":"d190b2f2-f6a2-405d-b1a7-1d39f4fc635c","Type":"ContainerDied","Data":"dd9a15ab5a048d85cbf006c4411a010aa16cfb4d2cd3a6110865076ca3d0111c"} Oct 30 22:35:43.995830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.994999 4305 scope.go:122] "RemoveContainer" containerID="6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193" Oct 30 22:35:43.995830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:43.995166 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx" Oct 30 22:35:44.050396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:44.050279 4305 scope.go:122] "RemoveContainer" containerID="6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193" Oct 30 22:35:44.063202 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:35:44.060956 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193\": container with ID starting with 6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193 not found: ID does not exist" containerID="6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193" Oct 30 22:35:44.063202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:44.061007 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193"} err="failed to get container status \"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193\": rpc error: code = NotFound desc = could not find container \"6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193\": container with ID starting with 6be38a3a005b78c8b70d78acb897a9ccabdadd4f413e51d491b0b47317d28193 not found: ID does not exist" Oct 30 22:35:44.181130 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:44.181096 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:35:44.185508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:44.185483 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-721c75-5d9bdb5dcb-t68kx"] Oct 30 22:35:45.520975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:45.520945 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" path="/var/lib/kubelet/pods/d190b2f2-f6a2-405d-b1a7-1d39f4fc635c/volumes" Oct 30 22:35:46.110029 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:35:46.109997 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:35:59.123428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.123391 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:35:59.124024 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.123596 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" podUID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" containerName="maas-api" containerID="cri-o://6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b" gracePeriod=30 Oct 30 22:35:59.487993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.487969 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:35:59.537251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.537219 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\" (UniqueName: \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\") pod \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " Oct 30 22:35:59.537414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.537356 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") pod \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " Oct 30 22:35:59.537414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.537394 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\") pod \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " Oct 30 22:35:59.537707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.537684 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\") pod \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\" (UID: \"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd\") " Oct 30 22:35:59.537831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.537790 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca" pod "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" (UID: "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:59.538318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.538292 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.539512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.539484 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls" pod "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" (UID: "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:59.539602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.539515 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz" pod "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" (UID: "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd"). InnerVolumeSpecName "kube-api-access-gjzkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:59.539703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.539685 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs" pod "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" (UID: "e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:59.795919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.751583 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.806543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.802964 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzkz\" (UniqueName: \"kubernetes.io/projected/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-kube-api-access-gjzkz\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:59.806543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:35:59.803015 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:00.296820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.284554 4305 generic.go:356] "Generic (PLEG): container finished" podID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" containerID="6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b" exitCode=0 Oct 30 22:36:00.296820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.284717 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" event={"ID":"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd","Type":"ContainerDied","Data":"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b"} Oct 30 22:36:00.296820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.284858 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" event={"ID":"e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd","Type":"ContainerDied","Data":"4c2fa22361a8e3baac3f6847a81656d73aaf4563d98af034b6f647de8cdebe69"} Oct 30 22:36:00.296820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.284895 4305 scope.go:122] "RemoveContainer" containerID="6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b" Oct 30 22:36:00.296820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.285071 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz" Oct 30 22:36:00.342254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.342225 4305 scope.go:122] "RemoveContainer" containerID="6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b" Oct 30 22:36:00.342713 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:00.342685 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b\": container with ID starting with 6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b not found: ID does not exist" containerID="6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b" Oct 30 22:36:00.342812 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.342722 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b"} err="failed to get container status \"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b\": rpc error: code = NotFound desc = could not find container \"6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b\": container with ID starting with 6f991d1993e7558d248ceb9fc7baf2e4d8516d24bc4a98b1dae400e3aad9e42b not found: ID does not exist" Oct 30 22:36:00.442738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.442702 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:36:00.482282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:00.482247 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-71c235-559d4fc7c-6dxrz"] Oct 30 22:36:01.533557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:01.533523 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" path="/var/lib/kubelet/pods/e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd/volumes" Oct 30 22:36:01.611268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:01.611231 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:36:01.611465 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:01.611438 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" podUID="e4867f6d-f77d-437f-9df5-eac91d93b67b" containerName="maas-api" containerID="cri-o://4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1" gracePeriod=30 Oct 30 22:36:01.995447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:01.995417 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:36:02.060062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.059960 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\") pod \"e4867f6d-f77d-437f-9df5-eac91d93b67b\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " Oct 30 22:36:02.060062 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.060038 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") pod \"e4867f6d-f77d-437f-9df5-eac91d93b67b\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " Oct 30 22:36:02.060299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.060107 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\") pod \"e4867f6d-f77d-437f-9df5-eac91d93b67b\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " Oct 30 22:36:02.060299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.060200 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\" (UniqueName: \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\") pod \"e4867f6d-f77d-437f-9df5-eac91d93b67b\" (UID: \"e4867f6d-f77d-437f-9df5-eac91d93b67b\") " Oct 30 22:36:02.061532 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.061501 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca" pod "e4867f6d-f77d-437f-9df5-eac91d93b67b" (UID: "e4867f6d-f77d-437f-9df5-eac91d93b67b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:02.065410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.065382 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6" pod "e4867f6d-f77d-437f-9df5-eac91d93b67b" (UID: "e4867f6d-f77d-437f-9df5-eac91d93b67b"). InnerVolumeSpecName "kube-api-access-27qg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:02.065923 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.065867 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls" pod "e4867f6d-f77d-437f-9df5-eac91d93b67b" (UID: "e4867f6d-f77d-437f-9df5-eac91d93b67b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:02.069029 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.069000 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs" pod "e4867f6d-f77d-437f-9df5-eac91d93b67b" (UID: "e4867f6d-f77d-437f-9df5-eac91d93b67b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:02.162760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.162719 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4867f6d-f77d-437f-9df5-eac91d93b67b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:02.162760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.162751 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:02.162760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.162761 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4867f6d-f77d-437f-9df5-eac91d93b67b-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:02.162760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.162771 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qg6\" (UniqueName: \"kubernetes.io/projected/e4867f6d-f77d-437f-9df5-eac91d93b67b-kube-api-access-27qg6\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:02.364661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.364570 4305 generic.go:356] "Generic (PLEG): container finished" podID="e4867f6d-f77d-437f-9df5-eac91d93b67b" containerID="4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1" exitCode=0 Oct 30 22:36:02.365104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.365070 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" event={"ID":"e4867f6d-f77d-437f-9df5-eac91d93b67b","Type":"ContainerDied","Data":"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1"} Oct 30 22:36:02.365353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.365315 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" event={"ID":"e4867f6d-f77d-437f-9df5-eac91d93b67b","Type":"ContainerDied","Data":"1581c7a94dd23a2b3243605c0e964a0872e452b51e86356b53f151613426f36d"} Oct 30 22:36:02.365546 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.365529 4305 scope.go:122] "RemoveContainer" containerID="4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1" Oct 30 22:36:02.365768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.365753 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp" Oct 30 22:36:02.439573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.438697 4305 scope.go:122] "RemoveContainer" containerID="4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1" Oct 30 22:36:02.439573 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:02.439228 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1\": container with ID starting with 4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1 not found: ID does not exist" containerID="4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1" Oct 30 22:36:02.439573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.439267 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1"} err="failed to get container status \"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1\": rpc error: code = NotFound desc = could not find container \"4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1\": container with ID starting with 4f94a260eb134386a3881b368973d0d2548cc642baaf3db65b2ce544a89a54f1 not found: ID does not exist" Oct 30 22:36:02.555812 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.555762 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:36:02.602398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:02.602365 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-73735c-7bf58b4667-zgfcp"] Oct 30 22:36:03.520886 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:03.520852 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4867f6d-f77d-437f-9df5-eac91d93b67b" path="/var/lib/kubelet/pods/e4867f6d-f77d-437f-9df5-eac91d93b67b/volumes" Oct 30 22:36:16.110914 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:16.110876 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:36:19.815671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:19.815645 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:36:19.817345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:19.817317 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:36:21.706159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:21.706124 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:36:26.047700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:26.047661 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:36:26.048207 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:26.047877 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" containerID="cri-o://b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26" gracePeriod=30 Oct 30 22:36:26.533699 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:26.533657 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:36:26.623641 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:26.623610 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:36:26.706960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:26.706927 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" podUID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" containerName="payload-processing" containerID="cri-o://414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19" gracePeriod=25 Oct 30 22:36:27.178363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.178327 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:36:27.206928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.206901 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\" (UniqueName: \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\") pod \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " Oct 30 22:36:27.207091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.207022 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\") pod \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\" (UID: \"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9\") " Oct 30 22:36:27.207365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.207341 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume" pod "1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" (UID: "1b86cdbb-380a-45dd-8abf-9a7e4bd429b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:27.207632 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.207613 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:27.209065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.209030 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl" pod "1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" (UID: "1b86cdbb-380a-45dd-8abf-9a7e4bd429b9"). InnerVolumeSpecName "kube-api-access-ss9dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:27.308545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.308514 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9dl\" (UniqueName: \"kubernetes.io/projected/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9-kube-api-access-ss9dl\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:27.628174 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.628145 4305 generic.go:356] "Generic (PLEG): container finished" podID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" containerID="414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19" exitCode=0 Oct 30 22:36:27.628334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.628191 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" Oct 30 22:36:27.628334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.628239 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" event={"ID":"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9","Type":"ContainerDied","Data":"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19"} Oct 30 22:36:27.628334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.628290 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm" event={"ID":"1b86cdbb-380a-45dd-8abf-9a7e4bd429b9","Type":"ContainerDied","Data":"94ed4ab08215723575cb39120cc769a4c777eb4cf2f26cc81c913cf0724f24f5"} Oct 30 22:36:27.628334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.628311 4305 scope.go:122] "RemoveContainer" containerID="414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19" Oct 30 22:36:27.639124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.639104 4305 scope.go:122] "RemoveContainer" containerID="414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19" Oct 30 22:36:27.639392 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:27.639376 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19\": container with ID starting with 414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19 not found: ID does not exist" containerID="414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19" Oct 30 22:36:27.639441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.639399 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19"} err="failed to get container status \"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19\": rpc error: code = NotFound desc = could not find container \"414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19\": container with ID starting with 414e0e09e46b79b6026cc62bc32207cffccb7fce3121e95dbc990eee4a8cfe19 not found: ID does not exist" Oct 30 22:36:27.785096 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.784980 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:36:27.830923 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.830897 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-6116bd-57d5945767-7jkvm"] Oct 30 22:36:27.976824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.976384 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:27.976824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:27.976457 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:29.524994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:29.524947 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" path="/var/lib/kubelet/pods/1b86cdbb-380a-45dd-8abf-9a7e4bd429b9/volumes" Oct 30 22:36:31.322482 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.322387 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:36:31.377248 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377207 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377418 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377316 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377418 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377402 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377556 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377459 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377549 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.377611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377562 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377609 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377689 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377816 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377908 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377869 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\") pod \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\" (UID: \"90349c34-9110-46f4-9cb1-a2a6f37b8f33\") " Oct 30 22:36:31.377959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.377895 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.378202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378149 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.378202 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378150 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.378409 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378268 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:31.378562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378521 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.378562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378547 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.378562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378560 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.378901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378574 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.378901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.378587 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.380206 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.380170 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "kube-api-access-ktwts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.380406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.380381 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:31.381169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.381142 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.381710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.381655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy" pod "90349c34-9110-46f4-9cb1-a2a6f37b8f33" (UID: "90349c34-9110-46f4-9cb1-a2a6f37b8f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.479702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.479665 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.479702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.479696 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.479702 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.479706 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktwts\" (UniqueName: \"kubernetes.io/projected/90349c34-9110-46f4-9cb1-a2a6f37b8f33-kube-api-access-ktwts\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.480027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.479719 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90349c34-9110-46f4-9cb1-a2a6f37b8f33-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.536510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.536461 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" podUID="8192c484-3f3f-4025-aa8c-93498cb87f93" containerName="payload-processing" containerID="cri-o://331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c" gracePeriod=25 Oct 30 22:36:31.624418 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.624304 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" podUID="23999a27-4ae7-4fc9-878d-4cccce9a4632" containerName="payload-pre-processing" containerID="cri-o://24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987" gracePeriod=25 Oct 30 22:36:31.658301 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.658269 4305 generic.go:356] "Generic (PLEG): container finished" podID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerID="b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26" exitCode=0 Oct 30 22:36:31.658458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.658330 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" Oct 30 22:36:31.658458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.658349 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" event={"ID":"90349c34-9110-46f4-9cb1-a2a6f37b8f33","Type":"ContainerDied","Data":"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26"} Oct 30 22:36:31.658458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.658441 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc" event={"ID":"90349c34-9110-46f4-9cb1-a2a6f37b8f33","Type":"ContainerDied","Data":"c93f78e710b8e5ce5911c06662c6936bc9cae96a1749a43bda5515d7f0a17cea"} Oct 30 22:36:31.658568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.658469 4305 scope.go:122] "RemoveContainer" containerID="b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26" Oct 30 22:36:31.744776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.744751 4305 scope.go:122] "RemoveContainer" containerID="b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26" Oct 30 22:36:31.745209 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:31.745179 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26\": container with ID starting with b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26 not found: ID does not exist" containerID="b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26" Oct 30 22:36:31.745294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.745221 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26"} err="failed to get container status \"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26\": rpc error: code = NotFound desc = could not find container \"b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26\": container with ID starting with b5efd5d41d0fb6c3c4092b334a92eecfea37e24187d8d574b4e90dfeeb961a26 not found: ID does not exist" Oct 30 22:36:31.825681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.825625 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:36:31.832032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.831993 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-fvlmc"] Oct 30 22:36:31.895847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.895779 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:36:31.961345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.961321 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:36:31.987858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.987828 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\") pod \"23999a27-4ae7-4fc9-878d-4cccce9a4632\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " Oct 30 22:36:31.988043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.987914 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\") pod \"8192c484-3f3f-4025-aa8c-93498cb87f93\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " Oct 30 22:36:31.988138 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988103 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\" (UniqueName: \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\") pod \"23999a27-4ae7-4fc9-878d-4cccce9a4632\" (UID: \"23999a27-4ae7-4fc9-878d-4cccce9a4632\") " Oct 30 22:36:31.988199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988141 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\" (UniqueName: \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\") pod \"8192c484-3f3f-4025-aa8c-93498cb87f93\" (UID: \"8192c484-3f3f-4025-aa8c-93498cb87f93\") " Oct 30 22:36:31.988199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988177 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume" pod "23999a27-4ae7-4fc9-878d-4cccce9a4632" (UID: "23999a27-4ae7-4fc9-878d-4cccce9a4632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:31.988296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988250 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume" pod "8192c484-3f3f-4025-aa8c-93498cb87f93" (UID: "8192c484-3f3f-4025-aa8c-93498cb87f93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:31.988728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988704 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23999a27-4ae7-4fc9-878d-4cccce9a4632-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.988867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.988733 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8192c484-3f3f-4025-aa8c-93498cb87f93-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.990544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.990515 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2" pod "23999a27-4ae7-4fc9-878d-4cccce9a4632" (UID: "23999a27-4ae7-4fc9-878d-4cccce9a4632"). InnerVolumeSpecName "kube-api-access-88sh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.990712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:31.990688 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2" pod "8192c484-3f3f-4025-aa8c-93498cb87f93" (UID: "8192c484-3f3f-4025-aa8c-93498cb87f93"). InnerVolumeSpecName "kube-api-access-6l4j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:32.090270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.090236 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88sh2\" (UniqueName: \"kubernetes.io/projected/23999a27-4ae7-4fc9-878d-4cccce9a4632-kube-api-access-88sh2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.090270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.090269 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l4j2\" (UniqueName: \"kubernetes.io/projected/8192c484-3f3f-4025-aa8c-93498cb87f93-kube-api-access-6l4j2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.666010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.665976 4305 generic.go:356] "Generic (PLEG): container finished" podID="8192c484-3f3f-4025-aa8c-93498cb87f93" containerID="331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c" exitCode=0 Oct 30 22:36:32.666419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.666025 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" Oct 30 22:36:32.666419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.666065 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" event={"ID":"8192c484-3f3f-4025-aa8c-93498cb87f93","Type":"ContainerDied","Data":"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c"} Oct 30 22:36:32.666419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.666122 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb" event={"ID":"8192c484-3f3f-4025-aa8c-93498cb87f93","Type":"ContainerDied","Data":"b78f270e4b9643f116fbe23f3337a5dc3f7962dc4b46b7a298e94cd459d490bb"} Oct 30 22:36:32.666419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.666155 4305 scope.go:122] "RemoveContainer" containerID="331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c" Oct 30 22:36:32.668697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.668671 4305 generic.go:356] "Generic (PLEG): container finished" podID="23999a27-4ae7-4fc9-878d-4cccce9a4632" containerID="24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987" exitCode=0 Oct 30 22:36:32.668839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.668711 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" Oct 30 22:36:32.668839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.668723 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" event={"ID":"23999a27-4ae7-4fc9-878d-4cccce9a4632","Type":"ContainerDied","Data":"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987"} Oct 30 22:36:32.668839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.668763 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp" event={"ID":"23999a27-4ae7-4fc9-878d-4cccce9a4632","Type":"ContainerDied","Data":"95671965c002fa59d886c6c76ab8d8a1ee32a4831ac53deea7ab961480e0eab6"} Oct 30 22:36:32.677956 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.677933 4305 scope.go:122] "RemoveContainer" containerID="331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c" Oct 30 22:36:32.678255 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:32.678225 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c\": container with ID starting with 331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c not found: ID does not exist" containerID="331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c" Oct 30 22:36:32.678369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.678265 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c"} err="failed to get container status \"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c\": rpc error: code = NotFound desc = could not find container \"331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c\": container with ID starting with 331ab6f0a6c33c5d9d8380e760bf4ca108a55bb75a78cc83f9f2b7fb1e3eb17c not found: ID does not exist" Oct 30 22:36:32.678369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.678290 4305 scope.go:122] "RemoveContainer" containerID="24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987" Oct 30 22:36:32.689535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.689514 4305 scope.go:122] "RemoveContainer" containerID="24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987" Oct 30 22:36:32.689822 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:32.689775 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987\": container with ID starting with 24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987 not found: ID does not exist" containerID="24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987" Oct 30 22:36:32.689822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.689814 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987"} err="failed to get container status \"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987\": rpc error: code = NotFound desc = could not find container \"24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987\": container with ID starting with 24e9b4a5fcb718218e6604d18bd287c566518ac8cc978ecfc5ed96d3245d4987 not found: ID does not exist" Oct 30 22:36:32.826048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.826012 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:36:32.829920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.829896 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d57980-644975995-869mp"] Oct 30 22:36:32.933742 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.933672 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:36:32.938271 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:32.938234 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d57980-6948d6646c-2hpbb"] Oct 30 22:36:33.519847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:33.519816 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23999a27-4ae7-4fc9-878d-4cccce9a4632" path="/var/lib/kubelet/pods/23999a27-4ae7-4fc9-878d-4cccce9a4632/volumes" Oct 30 22:36:33.520274 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:33.520259 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8192c484-3f3f-4025-aa8c-93498cb87f93" path="/var/lib/kubelet/pods/8192c484-3f3f-4025-aa8c-93498cb87f93/volumes" Oct 30 22:36:33.520703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:33.520689 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" path="/var/lib/kubelet/pods/90349c34-9110-46f4-9cb1-a2a6f37b8f33/volumes" Oct 30 22:36:46.111210 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:36:46.111174 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:36:50.665393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:50.665360 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:36:50.665784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:50.665605 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" containerName="maas-api" containerID="cri-o://f81d8afd277211f629e56e625f310e4b9b245afa791b03e68fb989e24bddd1af" gracePeriod=30 Oct 30 22:36:50.794229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:50.794190 4305 generic.go:356] "Generic (PLEG): container finished" podID="e68995b7-5379-48de-b2f2-376b8196a77a" containerID="f81d8afd277211f629e56e625f310e4b9b245afa791b03e68fb989e24bddd1af" exitCode=0 Oct 30 22:36:50.794334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:50.794259 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" event={"ID":"e68995b7-5379-48de-b2f2-376b8196a77a","Type":"ContainerDied","Data":"f81d8afd277211f629e56e625f310e4b9b245afa791b03e68fb989e24bddd1af"} Oct 30 22:36:51.113146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.113085 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:36:51.179505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.179464 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\") pod \"e68995b7-5379-48de-b2f2-376b8196a77a\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " Oct 30 22:36:51.179709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.179685 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\" (UniqueName: \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\") pod \"e68995b7-5379-48de-b2f2-376b8196a77a\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " Oct 30 22:36:51.179817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.179724 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\") pod \"e68995b7-5379-48de-b2f2-376b8196a77a\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " Oct 30 22:36:51.179817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.179758 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") pod \"e68995b7-5379-48de-b2f2-376b8196a77a\" (UID: \"e68995b7-5379-48de-b2f2-376b8196a77a\") " Oct 30 22:36:51.179952 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.179915 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca" pod "e68995b7-5379-48de-b2f2-376b8196a77a" (UID: "e68995b7-5379-48de-b2f2-376b8196a77a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:51.180478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.180454 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68995b7-5379-48de-b2f2-376b8196a77a-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:51.183189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.183156 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th" pod "e68995b7-5379-48de-b2f2-376b8196a77a" (UID: "e68995b7-5379-48de-b2f2-376b8196a77a"). InnerVolumeSpecName "kube-api-access-7j9th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:51.183412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.183356 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs" pod "e68995b7-5379-48de-b2f2-376b8196a77a" (UID: "e68995b7-5379-48de-b2f2-376b8196a77a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:51.183534 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.183507 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls" pod "e68995b7-5379-48de-b2f2-376b8196a77a" (UID: "e68995b7-5379-48de-b2f2-376b8196a77a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:51.281904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.281764 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9th\" (UniqueName: \"kubernetes.io/projected/e68995b7-5379-48de-b2f2-376b8196a77a-kube-api-access-7j9th\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:51.281904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.281813 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:51.281904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.281829 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68995b7-5379-48de-b2f2-376b8196a77a-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:51.803362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.803290 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" Oct 30 22:36:51.803362 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.803316 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" event={"ID":"e68995b7-5379-48de-b2f2-376b8196a77a","Type":"ContainerDied","Data":"ad9b6e0f7db34a13b75ea8a6c68ad21e80e6a3e2d63e7b4bb919f40253e6fcfc"} Oct 30 22:36:51.803836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.803389 4305 scope.go:122] "RemoveContainer" containerID="f81d8afd277211f629e56e625f310e4b9b245afa791b03e68fb989e24bddd1af" Oct 30 22:36:51.958647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:51.958614 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:36:52.004483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:52.004447 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79"] Oct 30 22:36:53.521158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:53.521125 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" path="/var/lib/kubelet/pods/e68995b7-5379-48de-b2f2-376b8196a77a/volumes" Oct 30 22:36:53.968671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:36:53.968629 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-7c5d18-54945c474c-clx79" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.91:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:37:00.316370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:00.316340 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:37:00.316911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:00.316535 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" containerID="cri-o://46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e" gracePeriod=30 Oct 30 22:37:05.607483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.607452 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:37:05.743441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743407 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.743577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743526 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.743638 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743608 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.743699 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743677 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.743771 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743725 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.743864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:05.743864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743837 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:05.743864 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.744040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.743971 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:05.744040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744027 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.744154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744056 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.744154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744074 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\") pod \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\" (UID: \"b9050aba-2a56-4fe3-8cdf-7abe2696bc74\") " Oct 30 22:37:05.744344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744301 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:05.744424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744352 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:05.744696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744678 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.744696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744697 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.744884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744707 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.744884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744716 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.744884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.744725 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.746021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.745997 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:05.746129 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.746074 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:05.746412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.746385 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "kube-api-access-j9nbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:05.746412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.746390 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo" pod "b9050aba-2a56-4fe3-8cdf-7abe2696bc74" (UID: "b9050aba-2a56-4fe3-8cdf-7abe2696bc74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:05.846162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.846080 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.846162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.846111 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.846162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.846122 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nbc\" (UniqueName: \"kubernetes.io/projected/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-kube-api-access-j9nbc\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.846162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.846133 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9050aba-2a56-4fe3-8cdf-7abe2696bc74-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:05.906731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.906699 4305 generic.go:356] "Generic (PLEG): container finished" podID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerID="46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e" exitCode=0 Oct 30 22:37:05.906926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.906743 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" Oct 30 22:37:05.906926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.906788 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerDied","Data":"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e"} Oct 30 22:37:05.906926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.906884 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm" event={"ID":"b9050aba-2a56-4fe3-8cdf-7abe2696bc74","Type":"ContainerDied","Data":"f3046cce3a9fdc60decaa8beaca9a9ec648b6b4d84d6369053024a12cdd96343"} Oct 30 22:37:05.906926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.906918 4305 scope.go:122] "RemoveContainer" containerID="46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e" Oct 30 22:37:05.930398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.930374 4305 scope.go:122] "RemoveContainer" containerID="05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c" Oct 30 22:37:05.939408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.939383 4305 scope.go:122] "RemoveContainer" containerID="46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e" Oct 30 22:37:05.939773 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:05.939744 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e\": container with ID starting with 46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e not found: ID does not exist" containerID="46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e" Oct 30 22:37:05.939894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.939777 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e"} err="failed to get container status \"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e\": rpc error: code = NotFound desc = could not find container \"46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e\": container with ID starting with 46f1aea231cdf73925494244b1bbbfac8d4916b87be1b3a22df00f4af195e67e not found: ID does not exist" Oct 30 22:37:05.939894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.939810 4305 scope.go:122] "RemoveContainer" containerID="05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c" Oct 30 22:37:05.940096 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:05.940068 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c\": container with ID starting with 05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c not found: ID does not exist" containerID="05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c" Oct 30 22:37:05.940155 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:05.940102 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c"} err="failed to get container status \"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c\": rpc error: code = NotFound desc = could not find container \"05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c\": container with ID starting with 05f96d2696429c3b119b358109ebfe11d9a4fdf04d128bd44d01c75a1825294c not found: ID does not exist" Oct 30 22:37:06.062134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.062102 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:37:06.066550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.066522 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7c5d18-openshift-default-5749d679f7-cqdnm"] Oct 30 22:37:06.622160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.622127 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:37:06.623776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.623749 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:06.624184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624163 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:06.624327 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624315 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" containerName="maas-api" Oct 30 22:37:06.624409 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624399 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" containerName="maas-api" Oct 30 22:37:06.624485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624476 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" containerName="maas-api" Oct 30 22:37:06.624560 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624550 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" containerName="maas-api" Oct 30 22:37:06.624647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624637 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" containerName="maas-api" Oct 30 22:37:06.624737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624726 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" containerName="maas-api" Oct 30 22:37:06.624896 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624875 4305 cpu_manager.go:414] "containerMap: removing container" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" Oct 30 22:37:06.624985 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.624975 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" Oct 30 22:37:06.625067 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625058 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" containerName="payload-processing" Oct 30 22:37:06.625137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625128 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" containerName="payload-processing" Oct 30 22:37:06.625216 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625207 4305 cpu_manager.go:414] "containerMap: removing container" podUID="23999a27-4ae7-4fc9-878d-4cccce9a4632" containerName="payload-pre-processing" Oct 30 22:37:06.625292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625281 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23999a27-4ae7-4fc9-878d-4cccce9a4632" containerName="payload-pre-processing" Oct 30 22:37:06.625404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625394 4305 cpu_manager.go:414] "containerMap: removing container" podUID="adefb7d0-117f-40a1-957b-e7b109de05d6" containerName="maas-api" Oct 30 22:37:06.625476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625466 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adefb7d0-117f-40a1-957b-e7b109de05d6" containerName="maas-api" Oct 30 22:37:06.625547 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625539 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e4867f6d-f77d-437f-9df5-eac91d93b67b" containerName="maas-api" Oct 30 22:37:06.625623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625610 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4867f6d-f77d-437f-9df5-eac91d93b67b" containerName="maas-api" Oct 30 22:37:06.625744 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.625730 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8192c484-3f3f-4025-aa8c-93498cb87f93" containerName="payload-processing" Oct 30 22:37:06.635027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635009 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8192c484-3f3f-4025-aa8c-93498cb87f93" containerName="payload-processing" Oct 30 22:37:06.635346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635325 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e736990e-f1c4-48c5-aef0-f0d7f4dc4cbd" containerName="maas-api" Oct 30 22:37:06.635557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635539 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="90349c34-9110-46f4-9cb1-a2a6f37b8f33" containerName="istio-proxy" Oct 30 22:37:06.635666 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635654 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b86cdbb-380a-45dd-8abf-9a7e4bd429b9" containerName="payload-processing" Oct 30 22:37:06.635821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635746 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="23999a27-4ae7-4fc9-878d-4cccce9a4632" containerName="payload-pre-processing" Oct 30 22:37:06.635936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.635924 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="adefb7d0-117f-40a1-957b-e7b109de05d6" containerName="maas-api" Oct 30 22:37:06.636088 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636075 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4867f6d-f77d-437f-9df5-eac91d93b67b" containerName="maas-api" Oct 30 22:37:06.636197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636185 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8192c484-3f3f-4025-aa8c-93498cb87f93" containerName="payload-processing" Oct 30 22:37:06.636276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636266 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:06.636361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636349 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d190b2f2-f6a2-405d-b1a7-1d39f4fc635c" containerName="maas-api" Oct 30 22:37:06.636442 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636432 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68995b7-5379-48de-b2f2-376b8196a77a" containerName="maas-api" Oct 30 22:37:06.636523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.636512 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:06.654238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.653161 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.703399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.703369 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d6b55f-serving-cert\"" Oct 30 22:37:06.727784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.714357 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:37:06.761225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.761192 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.761424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.761244 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.761424 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.761352 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsg2\" (UniqueName: \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.761555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.761445 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.862736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.862687 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsg2\" (UniqueName: \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.862968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.862888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.863050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.862980 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.863050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.863039 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.863293 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:06.863264 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-serving-cert: secret "maas-api-e2e-inf-a-d6b55f-serving-cert" not found Oct 30 22:37:06.863572 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:06.863359 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls podName:cc77f4d4-5742-45c6-a838-c316551364f5 nodeName:}" failed. No retries permitted until 2026-10-30 22:37:07.363336265 +0000 UTC m=+2147.958424566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls") pod "maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" (UID: "cc77f4d4-5742-45c6-a838-c316551364f5") : secret "maas-api-e2e-inf-a-d6b55f-serving-cert" not found Oct 30 22:37:06.863830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.863772 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.866238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.866212 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:06.954077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:06.952908 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsg2\" (UniqueName: \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:07.376887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:07.376777 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:07.379185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:07.379158 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6b55f-bf789b944-6z885\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:07.521001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:07.520967 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" path="/var/lib/kubelet/pods/b9050aba-2a56-4fe3-8cdf-7abe2696bc74/volumes" Oct 30 22:37:07.638090 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:07.638023 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:08.016045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:08.016008 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:37:08.020916 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:37:08.020889 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc77f4d4_5742_45c6_a838_c316551364f5.slice/crio-a59b341287892d7d0ff6c3e4c139d65bb25db5f0a19173a792663f45b85ddfaa WatchSource:0}: Error finding container a59b341287892d7d0ff6c3e4c139d65bb25db5f0a19173a792663f45b85ddfaa: Status 404 returned error can't find the container with id a59b341287892d7d0ff6c3e4c139d65bb25db5f0a19173a792663f45b85ddfaa Oct 30 22:37:08.022364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:08.022344 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:37:08.985376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:08.985341 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" event={"ID":"cc77f4d4-5742-45c6-a838-c316551364f5","Type":"ContainerStarted","Data":"a59b341287892d7d0ff6c3e4c139d65bb25db5f0a19173a792663f45b85ddfaa"} Oct 30 22:37:10.001530 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:10.001489 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" event={"ID":"cc77f4d4-5742-45c6-a838-c316551364f5","Type":"ContainerStarted","Data":"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e"} Oct 30 22:37:10.001958 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:10.001722 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:10.118962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:10.118905 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" podStartSLOduration=3.139429976 podStartE2EDuration="4.118890657s" totalImagesPullingTime="979.460681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:10.117450406 +0000 UTC m=+2150.712538712" watchObservedRunningTime="2026-10-30 22:37:10.118890657 +0000 UTC m=+2150.713978963" Oct 30 22:37:11.017836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:11.017785 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:37:11.018225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:11.017969 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" containerID="cri-o://8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791" gracePeriod=30 Oct 30 22:37:16.015866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.015824 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:37:16.110591 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:16.110555 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:37:16.335041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.335013 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:37:16.379994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.379965 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380017 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380045 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380133 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380101 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380151 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380224 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380249 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380338 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380373 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:16.380453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380433 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\") pod \"2c7804d7-e82e-4db3-b061-48de68a4db8e\" (UID: \"2c7804d7-e82e-4db3-b061-48de68a4db8e\") " Oct 30 22:37:16.380599 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380460 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:16.380599 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380478 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:16.380776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380747 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:16.380776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380756 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:16.381001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.380981 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.381076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.381002 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7804d7-e82e-4db3-b061-48de68a4db8e-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.381076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.381011 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.381076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.381019 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.381076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.381029 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.382633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.382607 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "kube-api-access-xwzzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:16.382757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.382644 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:16.382757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.382677 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:16.382757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.382712 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy" pod "2c7804d7-e82e-4db3-b061-48de68a4db8e" (UID: "2c7804d7-e82e-4db3-b061-48de68a4db8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:16.483060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.483031 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.483060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.483060 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.483228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.483073 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7804d7-e82e-4db3-b061-48de68a4db8e-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:16.483228 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:16.483086 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzzj\" (UniqueName: \"kubernetes.io/projected/2c7804d7-e82e-4db3-b061-48de68a4db8e-kube-api-access-xwzzj\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:17.053526 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.053494 4305 generic.go:356] "Generic (PLEG): container finished" podID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerID="8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791" exitCode=0 Oct 30 22:37:17.053968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.053558 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" Oct 30 22:37:17.053968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.053579 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" event={"ID":"2c7804d7-e82e-4db3-b061-48de68a4db8e","Type":"ContainerDied","Data":"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791"} Oct 30 22:37:17.053968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.053662 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf" event={"ID":"2c7804d7-e82e-4db3-b061-48de68a4db8e","Type":"ContainerDied","Data":"fa7e2a2616c37ecea3e46786a8306e9a1ee4e5c26e24bdf17b7efa8dacae8947"} Oct 30 22:37:17.053968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.053684 4305 scope.go:122] "RemoveContainer" containerID="8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791" Oct 30 22:37:17.067066 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.066973 4305 scope.go:122] "RemoveContainer" containerID="8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791" Oct 30 22:37:17.067432 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:17.067411 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791\": container with ID starting with 8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791 not found: ID does not exist" containerID="8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791" Oct 30 22:37:17.067517 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.067440 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791"} err="failed to get container status \"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791\": rpc error: code = NotFound desc = could not find container \"8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791\": container with ID starting with 8812506a42251fc4289313de073d06c248de474ff499695468e646bb05d5b791 not found: ID does not exist" Oct 30 22:37:17.213839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.213774 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:37:17.259658 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.259631 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-7tpgf"] Oct 30 22:37:17.520045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:17.520016 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" path="/var/lib/kubelet/pods/2c7804d7-e82e-4db3-b061-48de68a4db8e/volumes" Oct 30 22:37:29.961828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:29.934842 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:37:29.961828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:29.952533 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" podUID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" containerName="maas-api" containerID="cri-o://a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc" gracePeriod=30 Oct 30 22:37:30.399248 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.399209 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:37:30.543659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.543579 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\") pod \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " Oct 30 22:37:30.543815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.543693 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") pod \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " Oct 30 22:37:30.543877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.543789 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\") pod \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " Oct 30 22:37:30.543928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.543914 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\" (UniqueName: \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\") pod \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\" (UID: \"ea471d82-1606-4c1f-964f-b41b0dbcfda5\") " Oct 30 22:37:30.543985 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.543947 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca" pod "ea471d82-1606-4c1f-964f-b41b0dbcfda5" (UID: "ea471d82-1606-4c1f-964f-b41b0dbcfda5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:30.544728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.544700 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea471d82-1606-4c1f-964f-b41b0dbcfda5-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:30.546121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.546094 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs" pod "ea471d82-1606-4c1f-964f-b41b0dbcfda5" (UID: "ea471d82-1606-4c1f-964f-b41b0dbcfda5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:30.546254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.546231 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls" pod "ea471d82-1606-4c1f-964f-b41b0dbcfda5" (UID: "ea471d82-1606-4c1f-964f-b41b0dbcfda5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:30.546314 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.546276 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4" pod "ea471d82-1606-4c1f-964f-b41b0dbcfda5" (UID: "ea471d82-1606-4c1f-964f-b41b0dbcfda5"). InnerVolumeSpecName "kube-api-access-h5dw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:30.653854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.653788 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:30.653854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.653849 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea471d82-1606-4c1f-964f-b41b0dbcfda5-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:30.653854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:30.653860 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dw4\" (UniqueName: \"kubernetes.io/projected/ea471d82-1606-4c1f-964f-b41b0dbcfda5-kube-api-access-h5dw4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:31.214875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.214840 4305 generic.go:356] "Generic (PLEG): container finished" podID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" containerID="a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc" exitCode=0 Oct 30 22:37:31.214875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.214868 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" Oct 30 22:37:31.215405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.214913 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" event={"ID":"ea471d82-1606-4c1f-964f-b41b0dbcfda5","Type":"ContainerDied","Data":"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc"} Oct 30 22:37:31.215405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.214972 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf" event={"ID":"ea471d82-1606-4c1f-964f-b41b0dbcfda5","Type":"ContainerDied","Data":"8210e69bd71a845d5dab83cd9698ef57ab69d345b2db7efdcc3e38bc2c621a0e"} Oct 30 22:37:31.215405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.215004 4305 scope.go:122] "RemoveContainer" containerID="a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc" Oct 30 22:37:31.226010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.225990 4305 scope.go:122] "RemoveContainer" containerID="a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc" Oct 30 22:37:31.226320 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:31.226300 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc\": container with ID starting with a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc not found: ID does not exist" containerID="a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc" Oct 30 22:37:31.226384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.226331 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc"} err="failed to get container status \"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc\": rpc error: code = NotFound desc = could not find container \"a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc\": container with ID starting with a8336c8a6458dd89f444325b5d8abe652e3ecb59dbbd46925d8eb310579819cc not found: ID does not exist" Oct 30 22:37:31.445436 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.445396 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:37:31.449417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.449391 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6116bd-6464d994cc-5njhf"] Oct 30 22:37:31.521085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:31.520989 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" path="/var/lib/kubelet/pods/ea471d82-1606-4c1f-964f-b41b0dbcfda5/volumes" Oct 30 22:37:36.020764 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.020733 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:37:36.021250 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.021059 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" podUID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" containerName="maas-api" containerID="cri-o://b8b61df8059bddab30b3d3aef96d72af9822f35fc34c0e44fe5dca915b6838ae" gracePeriod=30 Oct 30 22:37:36.310219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.310140 4305 generic.go:356] "Generic (PLEG): container finished" podID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" containerID="b8b61df8059bddab30b3d3aef96d72af9822f35fc34c0e44fe5dca915b6838ae" exitCode=0 Oct 30 22:37:36.310386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.310213 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" event={"ID":"a096c4fd-0a25-4091-a461-6c03c58e6d1c","Type":"ContainerDied","Data":"b8b61df8059bddab30b3d3aef96d72af9822f35fc34c0e44fe5dca915b6838ae"} Oct 30 22:37:36.346175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.346148 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:37:36.467858 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.466670 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") pod \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " Oct 30 22:37:36.468912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.468885 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\") pod \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " Oct 30 22:37:36.469579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.469559 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\" (UniqueName: \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\") pod \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " Oct 30 22:37:36.477364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.471775 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\") pod \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\" (UID: \"a096c4fd-0a25-4091-a461-6c03c58e6d1c\") " Oct 30 22:37:36.477649 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.477618 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca" pod "a096c4fd-0a25-4091-a461-6c03c58e6d1c" (UID: "a096c4fd-0a25-4091-a461-6c03c58e6d1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:36.482550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.482526 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a096c4fd-0a25-4091-a461-6c03c58e6d1c-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:36.512308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.512249 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs" pod "a096c4fd-0a25-4091-a461-6c03c58e6d1c" (UID: "a096c4fd-0a25-4091-a461-6c03c58e6d1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:36.561014 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.560593 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s" pod "a096c4fd-0a25-4091-a461-6c03c58e6d1c" (UID: "a096c4fd-0a25-4091-a461-6c03c58e6d1c"). InnerVolumeSpecName "kube-api-access-gnb6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:36.561700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.561596 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls" pod "a096c4fd-0a25-4091-a461-6c03c58e6d1c" (UID: "a096c4fd-0a25-4091-a461-6c03c58e6d1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:36.584312 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.584278 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnb6s\" (UniqueName: \"kubernetes.io/projected/a096c4fd-0a25-4091-a461-6c03c58e6d1c-kube-api-access-gnb6s\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:36.584468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.584317 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:36.584468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:36.584333 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a096c4fd-0a25-4091-a461-6c03c58e6d1c-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:37.320267 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:37.320233 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" event={"ID":"a096c4fd-0a25-4091-a461-6c03c58e6d1c","Type":"ContainerDied","Data":"b8d940485a6e24f8bd91a13022b50ad22882e219e55352ffd3950951a344f47c"} Oct 30 22:37:37.320267 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:37.320266 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6" Oct 30 22:37:37.320731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:37.320302 4305 scope.go:122] "RemoveContainer" containerID="b8b61df8059bddab30b3d3aef96d72af9822f35fc34c0e44fe5dca915b6838ae" Oct 30 22:37:37.479933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:37.479899 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:37:37.524145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:37.524113 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a3600a-5f796b8576-wdrb6"] Oct 30 22:37:38.878786 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.878749 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:37:38.879585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879562 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" containerName="maas-api" Oct 30 22:37:38.879585 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879586 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" containerName="maas-api" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879611 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" containerName="maas-api" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879617 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" containerName="maas-api" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879657 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879665 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879700 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:38.879770 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879708 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9050aba-2a56-4fe3-8cdf-7abe2696bc74" containerName="istio-proxy" Oct 30 22:37:38.880118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879894 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" containerName="maas-api" Oct 30 22:37:38.880118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879914 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea471d82-1606-4c1f-964f-b41b0dbcfda5" containerName="maas-api" Oct 30 22:37:38.880118 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.879932 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7804d7-e82e-4db3-b061-48de68a4db8e" containerName="istio-proxy" Oct 30 22:37:38.884992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.884967 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.925644 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.925607 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:37:38.929771 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.929747 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.973452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.973415 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:37:38.974294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974227 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974281 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974313 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974355 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4x7\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974407 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlmd\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974439 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974473 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974494 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974512 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974619 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974678 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974700 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974722 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974737 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974839 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:38.974964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:38.974884 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.011486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.011451 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:37:39.076236 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076203 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076291 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.076437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076326 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.076437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076348 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076388 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4x7\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.076437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076437 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlmd\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076474 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076521 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076547 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076576 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076671 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.076735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076709 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076719 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076875 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076915 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076949 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.076976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077034 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077054 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077188 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077332 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077188 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077461 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077461 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077510 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.077792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077676 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.077792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.077755 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.079466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.079438 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.079578 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.079548 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.079640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.079616 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.079767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.079751 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.134989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.134912 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4x7\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.135132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.135017 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.135363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.135345 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlmd\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.135405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.135364 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.200017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.199975 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:39.264836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.248913 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:39.532278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.532237 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a096c4fd-0a25-4091-a461-6c03c58e6d1c" path="/var/lib/kubelet/pods/a096c4fd-0a25-4091-a461-6c03c58e6d1c/volumes" Oct 30 22:37:39.706870 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.706829 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:37:39.712344 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:37:39.712318 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode822d6f0_e475_4032_8e82_2901a5a52768.slice/crio-b7c2ef3e049594d3937cc22370c0d789c43e5f1c00ed21ebe67bc66bcd6e42c2 WatchSource:0}: Error finding container b7c2ef3e049594d3937cc22370c0d789c43e5f1c00ed21ebe67bc66bcd6e42c2: Status 404 returned error can't find the container with id b7c2ef3e049594d3937cc22370c0d789c43e5f1c00ed21ebe67bc66bcd6e42c2 Oct 30 22:37:39.715459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.715426 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.715544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.715495 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.715544 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.715524 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.759426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.759400 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:37:39.764313 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:37:39.764278 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045ebd6b_0a24_4584_9216_68e4f260adc8.slice/crio-80bfba2eec630cd557a2b4f61c25fae00b082d53c46214adb9e8d25e981375dc WatchSource:0}: Error finding container 80bfba2eec630cd557a2b4f61c25fae00b082d53c46214adb9e8d25e981375dc: Status 404 returned error can't find the container with id 80bfba2eec630cd557a2b4f61c25fae00b082d53c46214adb9e8d25e981375dc Oct 30 22:37:39.767368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.767333 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.767474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.767406 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.767474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:39.767436 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:40.385478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.385430 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" event={"ID":"045ebd6b-0a24-4584-9216-68e4f260adc8","Type":"ContainerStarted","Data":"3e2188a7bd5747a607e4a5d29d919520e4d30a61fa03aba79b9d7d391f9ebfce"} Oct 30 22:37:40.386099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.386050 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" event={"ID":"045ebd6b-0a24-4584-9216-68e4f260adc8","Type":"ContainerStarted","Data":"80bfba2eec630cd557a2b4f61c25fae00b082d53c46214adb9e8d25e981375dc"} Oct 30 22:37:40.415738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.412400 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" event={"ID":"e822d6f0-e475-4032-8e82-2901a5a52768","Type":"ContainerStarted","Data":"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621"} Oct 30 22:37:40.415738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.412486 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" event={"ID":"e822d6f0-e475-4032-8e82-2901a5a52768","Type":"ContainerStarted","Data":"b7c2ef3e049594d3937cc22370c0d789c43e5f1c00ed21ebe67bc66bcd6e42c2"} Oct 30 22:37:40.649661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.644354 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podStartSLOduration=2.64433337 podStartE2EDuration="2.64433337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:40.554395238 +0000 UTC m=+2181.149483547" watchObservedRunningTime="2026-10-30 22:37:40.64433337 +0000 UTC m=+2181.239421680" Oct 30 22:37:40.699819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:40.698906 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podStartSLOduration=2.698883002 podStartE2EDuration="2.698883002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:40.679092587 +0000 UTC m=+2181.274180894" watchObservedRunningTime="2026-10-30 22:37:40.698883002 +0000 UTC m=+2181.293971310" Oct 30 22:37:41.202626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.202435 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:41.203220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.203166 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:41.203348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.203240 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:41.270822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.263559 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:41.273981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.273938 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:41.274251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:41.274217 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:42.200821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:42.200691 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:42.200821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:42.200753 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:42.260824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:42.257790 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:42.260824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:42.257887 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:43.203873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:43.202664 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:43.203873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:43.202731 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:43.264821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:43.254374 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:43.264821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:43.254454 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:44.206964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:44.206832 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:44.206964 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:44.206898 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:44.267302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:44.267265 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:44.267878 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:44.267572 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:45.201191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:45.201155 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:45.201604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:45.201552 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:45.255245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:45.255206 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:45.257099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:45.257064 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:46.121377 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:46.121337 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:37:46.201494 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:46.201458 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:37:46.201706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:46.201515 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:37:46.254197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:46.254163 4305 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 22:37:46.254369 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:46.254214 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 22:37:47.188880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.188841 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:37:47.189686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.189647 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" containerID="cri-o://2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265" gracePeriod=30 Oct 30 22:37:47.207255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.207227 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:47.207542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.207511 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:47.208642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.208621 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:37:47.259555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.259527 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:47.259826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.259793 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:47.260443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:47.260422 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:37:52.478637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.478611 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:37:52.556706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.556672 4305 generic.go:356] "Generic (PLEG): container finished" podID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerID="2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265" exitCode=0 Oct 30 22:37:52.556919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.556723 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" Oct 30 22:37:52.556919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.556762 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerDied","Data":"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265"} Oct 30 22:37:52.556919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.556841 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx" event={"ID":"b7b667a3-896d-4485-a6aa-2cdc954aeb3f","Type":"ContainerDied","Data":"d06c49637703c0dfbd0f90de28ce4357e745393e37921d34812701db48d833fc"} Oct 30 22:37:52.556919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.556874 4305 scope.go:122] "RemoveContainer" containerID="2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265" Oct 30 22:37:52.568063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.568041 4305 scope.go:122] "RemoveContainer" containerID="d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637" Oct 30 22:37:52.577873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.577854 4305 scope.go:122] "RemoveContainer" containerID="2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265" Oct 30 22:37:52.578149 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:52.578130 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265\": container with ID starting with 2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265 not found: ID does not exist" containerID="2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265" Oct 30 22:37:52.578242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.578158 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265"} err="failed to get container status \"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265\": rpc error: code = NotFound desc = could not find container \"2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265\": container with ID starting with 2d6168a1fd85d5e4ddcbcf468054c97ad52c34c2f6d487b711a2376e717cb265 not found: ID does not exist" Oct 30 22:37:52.578242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.578175 4305 scope.go:122] "RemoveContainer" containerID="d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637" Oct 30 22:37:52.578462 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:37:52.578442 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637\": container with ID starting with d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637 not found: ID does not exist" containerID="d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637" Oct 30 22:37:52.578508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.578461 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637"} err="failed to get container status \"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637\": rpc error: code = NotFound desc = could not find container \"d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637\": container with ID starting with d4bfeebd55cef9fdfa2010adf24ecd3235c5409d7114db6b977c3cb245262637 not found: ID does not exist" Oct 30 22:37:52.594487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594464 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594560 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594595 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594711 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594722 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:52.594923 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594869 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594939 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.594983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594937 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:52.595093 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.594971 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:52.595093 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595061 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:52.595183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595134 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.595183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595169 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.595273 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595205 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\") pod \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\" (UID: \"b7b667a3-896d-4485-a6aa-2cdc954aeb3f\") " Oct 30 22:37:52.595578 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595558 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:52.595922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595902 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.596010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595926 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.596010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595942 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.596010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595955 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.596010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.595968 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.597076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.597048 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:52.597317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.597276 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:52.597403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.597350 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "kube-api-access-xj9pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:52.597571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.597554 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token" pod "b7b667a3-896d-4485-a6aa-2cdc954aeb3f" (UID: "b7b667a3-896d-4485-a6aa-2cdc954aeb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:52.708257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.706242 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj9pg\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-kube-api-access-xj9pg\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.708257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.706285 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.708257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.706299 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:52.708257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:52.706312 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b667a3-896d-4485-a6aa-2cdc954aeb3f-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:53.028040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:53.027951 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:37:53.035055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:53.035008 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d57980-openshift-default-94f4f6945-f2dxx"] Oct 30 22:37:53.528824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:37:53.527998 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" path="/var/lib/kubelet/pods/b7b667a3-896d-4485-a6aa-2cdc954aeb3f/volumes" Oct 30 22:38:00.217221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:00.217171 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:38:00.315829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:00.315778 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:38:05.218197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.218109 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" podUID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" containerName="payload-processing" containerID="cri-o://8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1" gracePeriod=25 Oct 30 22:38:05.316166 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.316094 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" podUID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" containerName="payload-pre-processing" containerID="cri-o://20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271" gracePeriod=25 Oct 30 22:38:05.502847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.502486 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:38:05.603080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.603055 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:38:05.620782 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.620754 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\" (UniqueName: \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\") pod \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " Oct 30 22:38:05.620955 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.620927 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\") pod \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\" (UID: \"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4\") " Oct 30 22:38:05.621381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.621346 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume" pod "5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" (UID: "5caadcd7-56b0-4e12-b49e-3cf9e7f679d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:05.621692 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.621668 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:05.623295 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.623267 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf" pod "5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" (UID: "5caadcd7-56b0-4e12-b49e-3cf9e7f679d4"). InnerVolumeSpecName "kube-api-access-ppjmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:05.645677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.645649 4305 generic.go:356] "Generic (PLEG): container finished" podID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" containerID="20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271" exitCode=0 Oct 30 22:38:05.645860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.645700 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" Oct 30 22:38:05.645860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.645733 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" event={"ID":"c311579e-2b1a-45c7-aa09-e6e0a6652d47","Type":"ContainerDied","Data":"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271"} Oct 30 22:38:05.645860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.645790 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt" event={"ID":"c311579e-2b1a-45c7-aa09-e6e0a6652d47","Type":"ContainerDied","Data":"2e6341b4ec64009a460d9d2393d6ff8786b08002b627c96bdcb80f546647a101"} Oct 30 22:38:05.645860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.645825 4305 scope.go:122] "RemoveContainer" containerID="20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271" Oct 30 22:38:05.647458 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.647434 4305 generic.go:356] "Generic (PLEG): container finished" podID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" containerID="8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1" exitCode=0 Oct 30 22:38:05.647565 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.647465 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" event={"ID":"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4","Type":"ContainerDied","Data":"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1"} Oct 30 22:38:05.647565 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.647489 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" Oct 30 22:38:05.647659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.647496 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6" event={"ID":"5caadcd7-56b0-4e12-b49e-3cf9e7f679d4","Type":"ContainerDied","Data":"5de9cf60a855b66d27d7e3d9cc71d529196baab3c021659896e495065e00738e"} Oct 30 22:38:05.664777 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.664755 4305 scope.go:122] "RemoveContainer" containerID="20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271" Oct 30 22:38:05.665239 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:05.665200 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271\": container with ID starting with 20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271 not found: ID does not exist" containerID="20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271" Oct 30 22:38:05.665419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.665364 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271"} err="failed to get container status \"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271\": rpc error: code = NotFound desc = could not find container \"20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271\": container with ID starting with 20116ef826261f6491405bdd372660f33bf2325a0f745757164e7cd079df3271 not found: ID does not exist" Oct 30 22:38:05.665419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.665402 4305 scope.go:122] "RemoveContainer" containerID="8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1" Oct 30 22:38:05.676650 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.676628 4305 scope.go:122] "RemoveContainer" containerID="8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1" Oct 30 22:38:05.676916 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:05.676899 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1\": container with ID starting with 8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1 not found: ID does not exist" containerID="8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1" Oct 30 22:38:05.676975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.676923 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1"} err="failed to get container status \"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1\": rpc error: code = NotFound desc = could not find container \"8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1\": container with ID starting with 8782c84e2caff9053cb6063476d1c23e7421c0344e71cce20a2b69eda211ceb1 not found: ID does not exist" Oct 30 22:38:05.722365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.722257 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\") pod \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " Oct 30 22:38:05.722708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.722668 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\" (UniqueName: \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\") pod \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\" (UID: \"c311579e-2b1a-45c7-aa09-e6e0a6652d47\") " Oct 30 22:38:05.722854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.722718 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume" pod "c311579e-2b1a-45c7-aa09-e6e0a6652d47" (UID: "c311579e-2b1a-45c7-aa09-e6e0a6652d47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:05.723593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.723568 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c311579e-2b1a-45c7-aa09-e6e0a6652d47-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:05.723711 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.723599 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjmf\" (UniqueName: \"kubernetes.io/projected/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4-kube-api-access-ppjmf\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:05.724947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.724923 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n" pod "c311579e-2b1a-45c7-aa09-e6e0a6652d47" (UID: "c311579e-2b1a-45c7-aa09-e6e0a6652d47"). InnerVolumeSpecName "kube-api-access-kvz8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:05.825011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.824978 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvz8n\" (UniqueName: \"kubernetes.io/projected/c311579e-2b1a-45c7-aa09-e6e0a6652d47-kube-api-access-kvz8n\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:05.828683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.828650 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:38:05.832409 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:05.832368 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b5313f-685744b7c7-4lhg6"] Oct 30 22:38:06.069339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:06.069250 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:38:06.073904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:06.073872 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b5313f-6679d85dcf-hfnzt"] Oct 30 22:38:07.519574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:07.519535 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" path="/var/lib/kubelet/pods/5caadcd7-56b0-4e12-b49e-3cf9e7f679d4/volumes" Oct 30 22:38:07.520097 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:07.520055 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" path="/var/lib/kubelet/pods/c311579e-2b1a-45c7-aa09-e6e0a6652d47/volumes" Oct 30 22:38:16.109990 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:16.109954 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:38:17.089701 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.089666 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:38:17.089925 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.089884 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" podUID="8046dff0-a710-4847-88df-0b4ece37203c" containerName="maas-api" containerID="cri-o://bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027" gracePeriod=30 Oct 30 22:38:17.572759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.572298 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:38:17.616425 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.616392 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\") pod \"8046dff0-a710-4847-88df-0b4ece37203c\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " Oct 30 22:38:17.616612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.616499 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\" (UniqueName: \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\") pod \"8046dff0-a710-4847-88df-0b4ece37203c\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " Oct 30 22:38:17.616612 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.616555 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") pod \"8046dff0-a710-4847-88df-0b4ece37203c\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " Oct 30 22:38:17.616854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.616791 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\") pod \"8046dff0-a710-4847-88df-0b4ece37203c\" (UID: \"8046dff0-a710-4847-88df-0b4ece37203c\") " Oct 30 22:38:17.617170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.617143 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca" pod "8046dff0-a710-4847-88df-0b4ece37203c" (UID: "8046dff0-a710-4847-88df-0b4ece37203c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:17.617454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.617437 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8046dff0-a710-4847-88df-0b4ece37203c-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:17.618756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.618728 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs" pod "8046dff0-a710-4847-88df-0b4ece37203c" (UID: "8046dff0-a710-4847-88df-0b4ece37203c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:17.618756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.618743 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls" pod "8046dff0-a710-4847-88df-0b4ece37203c" (UID: "8046dff0-a710-4847-88df-0b4ece37203c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:17.618933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.618755 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb" pod "8046dff0-a710-4847-88df-0b4ece37203c" (UID: "8046dff0-a710-4847-88df-0b4ece37203c"). InnerVolumeSpecName "kube-api-access-6jfnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:17.719158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.719117 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:17.719158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.719145 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfnb\" (UniqueName: \"kubernetes.io/projected/8046dff0-a710-4847-88df-0b4ece37203c-kube-api-access-6jfnb\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:17.719158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.719156 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8046dff0-a710-4847-88df-0b4ece37203c-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:17.745363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.745329 4305 generic.go:356] "Generic (PLEG): container finished" podID="8046dff0-a710-4847-88df-0b4ece37203c" containerID="bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027" exitCode=0 Oct 30 22:38:17.745555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.745392 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" Oct 30 22:38:17.745555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.745421 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" event={"ID":"8046dff0-a710-4847-88df-0b4ece37203c","Type":"ContainerDied","Data":"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027"} Oct 30 22:38:17.745555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.745509 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst" event={"ID":"8046dff0-a710-4847-88df-0b4ece37203c","Type":"ContainerDied","Data":"ad238846ef7f4090f88383c048dc55af8457d2ecb576d62cc2cf7476f30f3830"} Oct 30 22:38:17.745555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.745538 4305 scope.go:122] "RemoveContainer" containerID="bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027" Oct 30 22:38:17.756706 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.756687 4305 scope.go:122] "RemoveContainer" containerID="bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027" Oct 30 22:38:17.757039 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:17.757018 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027\": container with ID starting with bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027 not found: ID does not exist" containerID="bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027" Oct 30 22:38:17.757125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.757050 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027"} err="failed to get container status \"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027\": rpc error: code = NotFound desc = could not find container \"bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027\": container with ID starting with bbbe387faa3dbd2122fe46ca95966a62f939ba9690d6180780d94ef2ab24a027 not found: ID does not exist" Oct 30 22:38:17.911050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.911013 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:38:17.915753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:17.915725 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-bbd10f-77df8877f7-gbzst"] Oct 30 22:38:19.519276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:19.519239 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8046dff0-a710-4847-88df-0b4ece37203c" path="/var/lib/kubelet/pods/8046dff0-a710-4847-88df-0b4ece37203c/volumes" Oct 30 22:38:27.724833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.724757 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:38:27.725651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725614 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:27.725651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725644 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725682 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8046dff0-a710-4847-88df-0b4ece37203c" containerName="maas-api" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725690 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8046dff0-a710-4847-88df-0b4ece37203c" containerName="maas-api" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725721 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" containerName="payload-processing" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725729 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" containerName="payload-processing" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725750 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" containerName="payload-pre-processing" Oct 30 22:38:27.725814 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725758 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" containerName="payload-pre-processing" Oct 30 22:38:27.726060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725912 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:27.726060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725927 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5caadcd7-56b0-4e12-b49e-3cf9e7f679d4" containerName="payload-processing" Oct 30 22:38:27.726060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725935 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c311579e-2b1a-45c7-aa09-e6e0a6652d47" containerName="payload-pre-processing" Oct 30 22:38:27.726060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.725956 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8046dff0-a710-4847-88df-0b4ece37203c" containerName="maas-api" Oct 30 22:38:27.731002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.730979 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.780336 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.780307 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d585d2-openshift-default-dockercfg-q5shm\"" Oct 30 22:38:27.830538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.830502 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:38:27.840529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.840487 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.840529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.840527 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.840747 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.840624 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.840856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.840832 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.841037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.841011 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.841183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.841140 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.841302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.841274 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.841439 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.841343 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jkq\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.841502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.841476 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.957942 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958071 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958152 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958180 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jkq\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958246 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958268 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958293 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958315 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958718 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.958955 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.959691 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.960217 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.968822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.960421 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.987883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.986033 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:27.995140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:27.995104 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:28.037818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.036044 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jkq\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:28.037818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.036907 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\") pod \"e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:28.048171 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.046888 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:28.484407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.484175 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:38:28.488670 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:38:28.488643 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f12a95_569c_43a1_b631_e9991c57e11f.slice/crio-38113f49b07a3c3d141ec77bf37e55645ff4fb0b8712ef144cb9798cf6f94ece WatchSource:0}: Error finding container 38113f49b07a3c3d141ec77bf37e55645ff4fb0b8712ef144cb9798cf6f94ece: Status 404 returned error can't find the container with id 38113f49b07a3c3d141ec77bf37e55645ff4fb0b8712ef144cb9798cf6f94ece Oct 30 22:38:28.491571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.491536 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:28.491665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.491604 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:28.491665 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.491636 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:28.853164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.853080 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerStarted","Data":"a35cf403f47bb00621799616a80b187dd1b3275a630b23c6775a2a170e061978"} Oct 30 22:38:28.853164 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.853135 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerStarted","Data":"38113f49b07a3c3d141ec77bf37e55645ff4fb0b8712ef144cb9798cf6f94ece"} Oct 30 22:38:28.969275 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:28.969220 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podStartSLOduration=1.969205306 podStartE2EDuration="1.969205306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:28.966510038 +0000 UTC m=+2229.561598372" watchObservedRunningTime="2026-10-30 22:38:28.969205306 +0000 UTC m=+2229.564293684" Oct 30 22:38:29.052645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:29.052609 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:29.058968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:29.058942 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:29.859615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:29.859583 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:29.860575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:29.860553 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:38:31.701906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:31.701873 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:38:45.092829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:45.092785 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:38:46.110301 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:46.110267 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:38:46.724617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:46.724581 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" containerID="cri-o://79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9" gracePeriod=45 Oct 30 22:38:46.998742 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:46.998716 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:38:47.050472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.050444 4305 generic.go:356] "Generic (PLEG): container finished" podID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerID="79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9" exitCode=0 Oct 30 22:38:47.050626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.050488 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" Oct 30 22:38:47.050626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.050531 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" event={"ID":"2a27b5a8-9b9c-462b-b022-063b34a127b0","Type":"ContainerDied","Data":"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9"} Oct 30 22:38:47.050626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.050579 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" event={"ID":"2a27b5a8-9b9c-462b-b022-063b34a127b0","Type":"ContainerDied","Data":"2f366716be06a9c5c4bd1d839b60ad5823ae3ca3c46bc64d9f11e7a8870ade8c"} Oct 30 22:38:47.050626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.050597 4305 scope.go:122] "RemoveContainer" containerID="79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9" Oct 30 22:38:47.059205 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059178 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059298 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059388 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\" (UniqueName: \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059406 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.059474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059456 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059578 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059525 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059638 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\") pod \"2a27b5a8-9b9c-462b-b022-063b34a127b0\" (UID: \"2a27b5a8-9b9c-462b-b022-063b34a127b0\") " Oct 30 22:38:47.059788 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059758 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.059943 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.059914 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.060414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.060383 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.060414 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.060411 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.060562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.060428 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.060693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.060643 4305 scope.go:122] "RemoveContainer" containerID="79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9" Oct 30 22:38:47.061031 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:47.061004 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9\": container with ID starting with 79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9 not found: ID does not exist" containerID="79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9" Oct 30 22:38:47.061128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.061044 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9"} err="failed to get container status \"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9\": rpc error: code = NotFound desc = could not find container \"79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9\": container with ID starting with 79752cf8ef73224888d98d914298d5f74fbef31dcfbe231bff8311510e7681e9 not found: ID does not exist" Oct 30 22:38:47.061619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.061601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.061684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.061646 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "kube-api-access-th79s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:47.061865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.061792 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs" pod "2a27b5a8-9b9c-462b-b022-063b34a127b0" (UID: "2a27b5a8-9b9c-462b-b022-063b34a127b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:47.161620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.161583 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a27b5a8-9b9c-462b-b022-063b34a127b0-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.161620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.161618 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a27b5a8-9b9c-462b-b022-063b34a127b0-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.162226 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.161634 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th79s\" (UniqueName: \"kubernetes.io/projected/2a27b5a8-9b9c-462b-b022-063b34a127b0-kube-api-access-th79s\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.720821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.720556 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:38:47.768821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.768602 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc"] Oct 30 22:38:47.948099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:47.948039 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-2-w5-f80a0e-kserve-78c44655b5-fmcfc" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:38:49.519588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:49.519556 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" path="/var/lib/kubelet/pods/2a27b5a8-9b9c-462b-b022-063b34a127b0/volumes" Oct 30 22:38:54.065076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.065035 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:38:54.065609 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.065284 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" podUID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" containerName="maas-api" containerID="cri-o://061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa" gracePeriod=30 Oct 30 22:38:54.418772 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.418747 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:38:54.523912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.523871 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") pod \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " Oct 30 22:38:54.530181 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.530152 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\") pod \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " Oct 30 22:38:54.536297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.536268 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\" (UniqueName: \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\") pod \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " Oct 30 22:38:54.536523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.536503 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\") pod \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\" (UID: \"07847b61-406f-4ab5-9888-3d2ad97d1cc1\") " Oct 30 22:38:54.549841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.544990 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca" pod "07847b61-406f-4ab5-9888-3d2ad97d1cc1" (UID: "07847b61-406f-4ab5-9888-3d2ad97d1cc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:54.569212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.567102 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls" pod "07847b61-406f-4ab5-9888-3d2ad97d1cc1" (UID: "07847b61-406f-4ab5-9888-3d2ad97d1cc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:54.576862 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.575240 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx" pod "07847b61-406f-4ab5-9888-3d2ad97d1cc1" (UID: "07847b61-406f-4ab5-9888-3d2ad97d1cc1"). InnerVolumeSpecName "kube-api-access-2pvpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:54.579472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.579440 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs" pod "07847b61-406f-4ab5-9888-3d2ad97d1cc1" (UID: "07847b61-406f-4ab5-9888-3d2ad97d1cc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:54.639413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.639338 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvpx\" (UniqueName: \"kubernetes.io/projected/07847b61-406f-4ab5-9888-3d2ad97d1cc1-kube-api-access-2pvpx\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.639413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.639368 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07847b61-406f-4ab5-9888-3d2ad97d1cc1-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.639413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.639380 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:54.639623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:54.639414 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07847b61-406f-4ab5-9888-3d2ad97d1cc1-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:55.167815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.164925 4305 generic.go:356] "Generic (PLEG): container finished" podID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" containerID="061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa" exitCode=0 Oct 30 22:38:55.167815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.165164 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" event={"ID":"07847b61-406f-4ab5-9888-3d2ad97d1cc1","Type":"ContainerDied","Data":"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa"} Oct 30 22:38:55.167815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.165228 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" event={"ID":"07847b61-406f-4ab5-9888-3d2ad97d1cc1","Type":"ContainerDied","Data":"03011dfdd3b01c53b37e4782326035fffb2aeff7a6c35473212002f4c374624b"} Oct 30 22:38:55.167815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.165259 4305 scope.go:122] "RemoveContainer" containerID="061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa" Oct 30 22:38:55.167815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.165450 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm" Oct 30 22:38:55.186987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.186838 4305 scope.go:122] "RemoveContainer" containerID="061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa" Oct 30 22:38:55.187192 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:55.187157 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa\": container with ID starting with 061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa not found: ID does not exist" containerID="061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa" Oct 30 22:38:55.187324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.187208 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa"} err="failed to get container status \"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa\": rpc error: code = NotFound desc = could not find container \"061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa\": container with ID starting with 061ca442cd500fce8dbe8596ab8d7bb4d2c51130d13c9219baa765cb130661aa not found: ID does not exist" Oct 30 22:38:55.326627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.326594 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:38:55.373150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.373118 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b5313f-7b4c4f598b-7qzzm"] Oct 30 22:38:55.520070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.520035 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" path="/var/lib/kubelet/pods/07847b61-406f-4ab5-9888-3d2ad97d1cc1/volumes" Oct 30 22:38:55.628555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.628521 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:38:55.629169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629145 4305 cpu_manager.go:414] "containerMap: removing container" podUID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" containerName="maas-api" Oct 30 22:38:55.629374 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629349 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" containerName="maas-api" Oct 30 22:38:55.629486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629442 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:55.629486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629452 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:55.629486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629467 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" Oct 30 22:38:55.629486 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629473 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" Oct 30 22:38:55.629645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629632 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a27b5a8-9b9c-462b-b022-063b34a127b0" containerName="main" Oct 30 22:38:55.629691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629650 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="07847b61-406f-4ab5-9888-3d2ad97d1cc1" containerName="maas-api" Oct 30 22:38:55.629691 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.629660 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b667a3-896d-4485-a6aa-2cdc954aeb3f" containerName="istio-proxy" Oct 30 22:38:55.634381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.634363 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.683261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.683228 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-771ff9-serving-cert\"" Oct 30 22:38:55.691789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.691758 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:38:55.758348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.758315 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.758529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.758385 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.758529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.758507 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56w4n\" (UniqueName: \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.758628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.758614 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.860312 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.860226 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.860506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.860366 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.860506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.860449 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.860632 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.860531 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w4n\" (UniqueName: \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.860632 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:55.860538 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-serving-cert: secret "maas-api-e2e-api-a-771ff9-serving-cert" not found Oct 30 22:38:55.860632 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:38:55.860620 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls podName:2d6f6d42-621c-4f0e-b194-118512ee0366 nodeName:}" failed. No retries permitted until 2026-10-30 22:38:56.360601107 +0000 UTC m=+2256.955689392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls") pod "maas-api-e2e-api-a-771ff9-5f4d546885-945t6" (UID: "2d6f6d42-621c-4f0e-b194-118512ee0366") : secret "maas-api-e2e-api-a-771ff9-serving-cert" not found Oct 30 22:38:55.861087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.861059 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.863028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.863003 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:55.938818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:55.933719 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w4n\" (UniqueName: \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:56.374920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:56.374885 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:56.377466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:56.377437 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") pod \"maas-api-e2e-api-a-771ff9-5f4d546885-945t6\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:56.548574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:56.548539 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:56.932875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:56.932837 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:38:56.938712 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:38:56.938684 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6f6d42_621c_4f0e_b194_118512ee0366.slice/crio-240dde4f7545d3905abf43722b499b550676521f5142d6f95e42aa2fef65c301 WatchSource:0}: Error finding container 240dde4f7545d3905abf43722b499b550676521f5142d6f95e42aa2fef65c301: Status 404 returned error can't find the container with id 240dde4f7545d3905abf43722b499b550676521f5142d6f95e42aa2fef65c301 Oct 30 22:38:57.180355 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:57.180319 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" event={"ID":"2d6f6d42-621c-4f0e-b194-118512ee0366","Type":"ContainerStarted","Data":"240dde4f7545d3905abf43722b499b550676521f5142d6f95e42aa2fef65c301"} Oct 30 22:38:58.188000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:58.187909 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" event={"ID":"2d6f6d42-621c-4f0e-b194-118512ee0366","Type":"ContainerStarted","Data":"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec"} Oct 30 22:38:58.188416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:58.188063 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:38:58.298808 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:38:58.298754 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" podStartSLOduration=2.433101278 podStartE2EDuration="3.298739086s" totalImagesPullingTime="865.637808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:58.297340478 +0000 UTC m=+2258.892428784" watchObservedRunningTime="2026-10-30 22:38:58.298739086 +0000 UTC m=+2258.893827391" Oct 30 22:39:00.114832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.114762 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" podUID="ecb2a136-901d-4936-9c35-fadb3df7307d" containerName="main" containerID="cri-o://4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7" gracePeriod=45 Oct 30 22:39:00.398904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.398876 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:39:00.545042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.544953 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545041 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\" (UniqueName: \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545319 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545111 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545319 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545189 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545319 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545266 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545392 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\") pod \"ecb2a136-901d-4936-9c35-fadb3df7307d\" (UID: \"ecb2a136-901d-4936-9c35-fadb3df7307d\") " Oct 30 22:39:00.545849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.545685 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:00.546046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.546007 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:00.546222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.546197 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:00.546538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.546490 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:00.546538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.546519 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:00.546538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.546534 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:00.548025 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.548000 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "kube-api-access-b4j7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:00.548126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.548056 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:00.548337 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.548315 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs" pod "ecb2a136-901d-4936-9c35-fadb3df7307d" (UID: "ecb2a136-901d-4936-9c35-fadb3df7307d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:00.648708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.648636 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb2a136-901d-4936-9c35-fadb3df7307d-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:00.648708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.648661 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb2a136-901d-4936-9c35-fadb3df7307d-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:00.648708 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:00.648671 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4j7d\" (UniqueName: \"kubernetes.io/projected/ecb2a136-901d-4936-9c35-fadb3df7307d-kube-api-access-b4j7d\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:01.208508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.208471 4305 generic.go:356] "Generic (PLEG): container finished" podID="ecb2a136-901d-4936-9c35-fadb3df7307d" containerID="4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7" exitCode=0 Oct 30 22:39:01.209015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.208516 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" Oct 30 22:39:01.209015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.208552 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" event={"ID":"ecb2a136-901d-4936-9c35-fadb3df7307d","Type":"ContainerDied","Data":"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7"} Oct 30 22:39:01.209015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.208597 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp" event={"ID":"ecb2a136-901d-4936-9c35-fadb3df7307d","Type":"ContainerDied","Data":"47e466aeb4bbdcf2465a38bb12b2bdd655a65393ebd858a47ced46f13b09ee78"} Oct 30 22:39:01.209015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.208613 4305 scope.go:122] "RemoveContainer" containerID="4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7" Oct 30 22:39:01.220366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.220333 4305 scope.go:122] "RemoveContainer" containerID="4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7" Oct 30 22:39:01.220782 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:01.220749 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7\": container with ID starting with 4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7 not found: ID does not exist" containerID="4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7" Oct 30 22:39:01.220960 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.220807 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7"} err="failed to get container status \"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7\": rpc error: code = NotFound desc = could not find container \"4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7\": container with ID starting with 4292a6e81e72b19e12ed26ffa74d995468ff77d1be56bde64384603c23531ed7 not found: ID does not exist" Oct 30 22:39:01.364555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.364518 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:39:01.410406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.410377 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-29bbe735/e2e-lifecycle-model-29bbe735-kserve-7d44968549-8rjhp"] Oct 30 22:39:01.522307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:01.522218 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb2a136-901d-4936-9c35-fadb3df7307d" path="/var/lib/kubelet/pods/ecb2a136-901d-4936-9c35-fadb3df7307d/volumes" Oct 30 22:39:03.197726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:03.197696 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:39:03.198173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:03.197915 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" containerID="cri-o://2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae" gracePeriod=30 Oct 30 22:39:04.200361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:04.200328 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:39:06.585906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:06.585866 4305 patch_prober.go:29] interesting pod/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:39:06.586288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:06.585929 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:39:08.466561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.466495 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:39:08.578792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.578757 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.578997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.578902 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579001 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579055 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579070 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.579107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579090 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579273 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579286 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:39:08.579348 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579328 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579366 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\") pod \"a27da875-b9a4-43b0-b969-4a439c63f31d\" (UID: \"a27da875-b9a4-43b0-b969-4a439c63f31d\") " Oct 30 22:39:08.579590 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579567 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.579737 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579686 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.579841 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.579780 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.580119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.580098 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.580210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.580124 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.580210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.580139 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.580210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.580152 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.580210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.580164 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27da875-b9a4-43b0-b969-4a439c63f31d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.581441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.581416 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:39:08.581441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.581433 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "kube-api-access-xs6w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:08.581680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.581667 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:08.581736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.581678 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy" pod "a27da875-b9a4-43b0-b969-4a439c63f31d" (UID: "a27da875-b9a4-43b0-b969-4a439c63f31d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:08.681627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.681578 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.681627 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.681619 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6w4\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-kube-api-access-xs6w4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.681912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.681636 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.681912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.681650 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27da875-b9a4-43b0-b969-4a439c63f31d-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:08.721768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.721695 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:39:08.727103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:08.727070 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:39:09.307826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.302322 4305 generic.go:356] "Generic (PLEG): container finished" podID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerID="2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae" exitCode=0 Oct 30 22:39:09.311826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.311296 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" Oct 30 22:39:09.329820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.316879 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerDied","Data":"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae"} Oct 30 22:39:09.329820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.316963 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx" event={"ID":"a27da875-b9a4-43b0-b969-4a439c63f31d","Type":"ContainerDied","Data":"82b5a7279b83edc554599efd03f52039916f22f746f51978fa88906e85d06bd5"} Oct 30 22:39:09.329820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.316995 4305 scope.go:122] "RemoveContainer" containerID="2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae" Oct 30 22:39:09.351604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.351574 4305 scope.go:122] "RemoveContainer" containerID="c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35" Oct 30 22:39:09.366947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.366917 4305 scope.go:122] "RemoveContainer" containerID="2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae" Oct 30 22:39:09.367536 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:09.367512 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae\": container with ID starting with 2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae not found: ID does not exist" containerID="2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae" Oct 30 22:39:09.367687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.367659 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae"} err="failed to get container status \"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae\": rpc error: code = NotFound desc = could not find container \"2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae\": container with ID starting with 2f6d66d0648d83abbbe62cdcc49d6ec91c3dd588eab5a783660c532b67767dae not found: ID does not exist" Oct 30 22:39:09.368040 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.368019 4305 scope.go:122] "RemoveContainer" containerID="c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35" Oct 30 22:39:09.368480 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:09.368460 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35\": container with ID starting with c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35 not found: ID does not exist" containerID="c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35" Oct 30 22:39:09.368630 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.368607 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35"} err="failed to get container status \"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35\": rpc error: code = NotFound desc = could not find container \"c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35\": container with ID starting with c4414f480ecebf2ffbf327a613ceefa88a74f9ad3dbb167bab9f75f60c357b35 not found: ID does not exist" Oct 30 22:39:09.587401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.587311 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:39:09.587401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:09.587364 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b5313f-openshift-default-6986998b99-kf6xx"] Oct 30 22:39:11.527058 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:11.524837 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" path="/var/lib/kubelet/pods/a27da875-b9a4-43b0-b969-4a439c63f31d/volumes" Oct 30 22:39:16.112705 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:16.112643 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:39:23.745384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:23.745329 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" containerID="cri-o://3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c" gracePeriod=45 Oct 30 22:39:23.748403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:23.748348 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" containerID="cri-o://1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217" gracePeriod=45 Oct 30 22:39:23.871876 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:23.871772 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused" Oct 30 22:39:24.021020 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.020988 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:39:24.052502 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.052474 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:39:24.085712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.085682 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.085883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.085746 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.085883 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.085815 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.085981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.085883 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\" (UniqueName: \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.085981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.085942 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.086072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086005 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.086072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086040 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.086072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086066 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.086233 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086099 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.086233 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086170 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.086233 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086210 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.086397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086237 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\") pod \"b1597862-e6b8-44b2-9c59-4cf83474e109\" (UID: \"b1597862-e6b8-44b2-9c59-4cf83474e109\") " Oct 30 22:39:24.086397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086241 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.086397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086263 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.086397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086308 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\" (UniqueName: \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\") pod \"08457a3e-e38f-4246-babd-2542900363c7\" (UID: \"08457a3e-e38f-4246-babd-2542900363c7\") " Oct 30 22:39:24.086620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086429 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.086620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086604 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.086736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.086627 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.087041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087001 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.087041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087028 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.087041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087042 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.087366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087057 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.087366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087072 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.087366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.087087 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.089000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.088948 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.089113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.089061 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "kube-api-access-dx99w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:24.089177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.089123 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs" pod "b1597862-e6b8-44b2-9c59-4cf83474e109" (UID: "b1597862-e6b8-44b2-9c59-4cf83474e109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:24.089365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.089336 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:24.089365 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.089344 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.089950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.089930 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh" pod "08457a3e-e38f-4246-babd-2542900363c7" (UID: "08457a3e-e38f-4246-babd-2542900363c7"). InnerVolumeSpecName "kube-api-access-mz2nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:24.188633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188599 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08457a3e-e38f-4246-babd-2542900363c7-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.188792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188632 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx99w\" (UniqueName: \"kubernetes.io/projected/b1597862-e6b8-44b2-9c59-4cf83474e109-kube-api-access-dx99w\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.188792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188656 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1597862-e6b8-44b2-9c59-4cf83474e109-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.188792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188665 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1597862-e6b8-44b2-9c59-4cf83474e109-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.188792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188674 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08457a3e-e38f-4246-babd-2542900363c7-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.188792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.188683 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2nh\" (UniqueName: \"kubernetes.io/projected/08457a3e-e38f-4246-babd-2542900363c7-kube-api-access-mz2nh\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.407427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.407331 4305 generic.go:356] "Generic (PLEG): container finished" podID="08457a3e-e38f-4246-babd-2542900363c7" containerID="3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c" exitCode=0 Oct 30 22:39:24.407427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.407382 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" Oct 30 22:39:24.407664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.407418 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" event={"ID":"08457a3e-e38f-4246-babd-2542900363c7","Type":"ContainerDied","Data":"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c"} Oct 30 22:39:24.407664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.407471 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v" event={"ID":"08457a3e-e38f-4246-babd-2542900363c7","Type":"ContainerDied","Data":"e35de184b06c8fd39e9e16a53cd5c2b851e22cd5fa2c84eda108404f07f77b68"} Oct 30 22:39:24.407664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.407494 4305 scope.go:122] "RemoveContainer" containerID="3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c" Oct 30 22:39:24.409038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.409017 4305 generic.go:356] "Generic (PLEG): container finished" podID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerID="1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217" exitCode=0 Oct 30 22:39:24.409150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.409057 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" event={"ID":"b1597862-e6b8-44b2-9c59-4cf83474e109","Type":"ContainerDied","Data":"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217"} Oct 30 22:39:24.409150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.409085 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" event={"ID":"b1597862-e6b8-44b2-9c59-4cf83474e109","Type":"ContainerDied","Data":"306eddf452144264037645bb6c149f6e2a7e46949e8a8e443e17e365c2477cae"} Oct 30 22:39:24.409150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.409060 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc" Oct 30 22:39:24.422242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.422224 4305 scope.go:122] "RemoveContainer" containerID="3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c" Oct 30 22:39:24.422574 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:24.422556 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c\": container with ID starting with 3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c not found: ID does not exist" containerID="3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c" Oct 30 22:39:24.422651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.422584 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c"} err="failed to get container status \"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c\": rpc error: code = NotFound desc = could not find container \"3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c\": container with ID starting with 3f16b5477521ea9d2ea7501f298226a9ff073da3bce2d71c63acc1c462d4c18c not found: ID does not exist" Oct 30 22:39:24.422651 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.422606 4305 scope.go:122] "RemoveContainer" containerID="1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217" Oct 30 22:39:24.431729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.431708 4305 scope.go:122] "RemoveContainer" containerID="1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217" Oct 30 22:39:24.432035 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:24.432010 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217\": container with ID starting with 1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217 not found: ID does not exist" containerID="1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217" Oct 30 22:39:24.432104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.432045 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217"} err="failed to get container status \"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217\": rpc error: code = NotFound desc = could not find container \"1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217\": container with ID starting with 1c6ce557a2270a9014a71962dc1670a39fc78cb779e6a34410d2012f69cd6217 not found: ID does not exist" Oct 30 22:39:24.562618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.562582 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:39:24.565775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.565747 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-2-w1-af30a9-kserve-54d6fd8497-2tn5v"] Oct 30 22:39:24.668046 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.667974 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:39:24.671059 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:24.671033 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/distinct-w1-af30a9-kserve-84c44cd7fd-wnklc"] Oct 30 22:39:25.519196 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:25.519161 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08457a3e-e38f-4246-babd-2542900363c7" path="/var/lib/kubelet/pods/08457a3e-e38f-4246-babd-2542900363c7/volumes" Oct 30 22:39:25.519715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:25.519700 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" path="/var/lib/kubelet/pods/b1597862-e6b8-44b2-9c59-4cf83474e109/volumes" Oct 30 22:39:33.939363 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939284 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:39:33.939877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939862 4305 cpu_manager.go:414] "containerMap: removing container" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939878 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939895 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939900 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939908 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ecb2a136-901d-4936-9c35-fadb3df7307d" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939913 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb2a136-901d-4936-9c35-fadb3df7307d" containerName="main" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939929 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.939931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939934 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939969 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.939977 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.940125 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.940151 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27da875-b9a4-43b0-b969-4a439c63f31d" containerName="istio-proxy" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.940165 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="08457a3e-e38f-4246-babd-2542900363c7" containerName="main" Oct 30 22:39:33.940179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.940175 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1597862-e6b8-44b2-9c59-4cf83474e109" containerName="main" Oct 30 22:39:33.940443 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.940182 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb2a136-901d-4936-9c35-fadb3df7307d" containerName="main" Oct 30 22:39:33.945390 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.945371 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:33.994478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:33.994447 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d585d2-serving-cert\"" Oct 30 22:39:34.031378 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.031341 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p47t\" (UniqueName: \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.031563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.031549 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.031615 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.031582 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.031771 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.031746 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.044286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.044260 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:39:34.133056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.133019 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.133246 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.133102 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p47t\" (UniqueName: \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.133246 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:34.133183 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-serving-cert: secret "maas-api-e2e-inf-b-d585d2-serving-cert" not found Oct 30 22:39:34.133246 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:34.133249 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls podName:be96707f-c6f0-4313-9543-fd586492afa5 nodeName:}" failed. No retries permitted until 2026-10-30 22:39:34.633230307 +0000 UTC m=+2295.228318593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls") pod "maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" (UID: "be96707f-c6f0-4313-9543-fd586492afa5") : secret "maas-api-e2e-inf-b-d585d2-serving-cert" not found Oct 30 22:39:34.133435 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.133331 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.133435 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.133370 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.134017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.133993 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.135891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.135873 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.217868 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.214547 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p47t\" (UniqueName: \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.646029 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.645941 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.648536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.648505 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") pod \"maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:34.858687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:34.858652 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:35.264223 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:35.264184 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:39:35.268564 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:39:35.268526 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe96707f_c6f0_4313_9543_fd586492afa5.slice/crio-5a3cde5e63c546eb8b7d646fdd34dcf62e50a8e51308760a68b5c1f40d92cdcf WatchSource:0}: Error finding container 5a3cde5e63c546eb8b7d646fdd34dcf62e50a8e51308760a68b5c1f40d92cdcf: Status 404 returned error can't find the container with id 5a3cde5e63c546eb8b7d646fdd34dcf62e50a8e51308760a68b5c1f40d92cdcf Oct 30 22:39:35.476404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:35.476365 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" event={"ID":"be96707f-c6f0-4313-9543-fd586492afa5","Type":"ContainerStarted","Data":"5a3cde5e63c546eb8b7d646fdd34dcf62e50a8e51308760a68b5c1f40d92cdcf"} Oct 30 22:39:36.484256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:36.484222 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" event={"ID":"be96707f-c6f0-4313-9543-fd586492afa5","Type":"ContainerStarted","Data":"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6"} Oct 30 22:39:36.484671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:36.484321 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:36.596392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:36.596334 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" podStartSLOduration=2.715503166 podStartE2EDuration="3.596318788s" totalImagesPullingTime="880.815622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:36.59443832 +0000 UTC m=+2297.189526651" watchObservedRunningTime="2026-10-30 22:39:36.596318788 +0000 UTC m=+2297.191407094" Oct 30 22:39:42.496886 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:42.496849 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:39:43.291890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.291858 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:39:43.296358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.296335 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.347731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.347701 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-afb98c-serving-cert\"" Oct 30 22:39:43.399358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.399320 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:39:43.472857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.472784 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdbb9\" (UniqueName: \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.473048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.473027 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.473132 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.473115 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.473192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.473168 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.577911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.574113 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdbb9\" (UniqueName: \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.577911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.574270 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.577911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.574343 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.577911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.574376 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.578698 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:43.578179 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-serving-cert: secret "maas-api-e2e-inf-b-afb98c-serving-cert" not found Oct 30 22:39:43.578698 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:43.578359 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls podName:088f3617-8a8e-490f-9bad-94bba1a95119 nodeName:}" failed. No retries permitted until 2026-10-30 22:39:44.07833435 +0000 UTC m=+2304.673422671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls") pod "maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" (UID: "088f3617-8a8e-490f-9bad-94bba1a95119") : secret "maas-api-e2e-inf-b-afb98c-serving-cert" not found Oct 30 22:39:43.607887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.587697 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.618852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.617471 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:43.637107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:43.637071 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdbb9\" (UniqueName: \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:44.082215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:44.082181 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:44.084629 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:44.084598 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") pod \"maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:44.210305 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:44.210273 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:44.589231 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:44.589196 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:39:44.594385 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:39:44.594348 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088f3617_8a8e_490f_9bad_94bba1a95119.slice/crio-770ead3d649b1a66a7da222e9a0b3f717a440039b249272c9bec4dcaf79c1bf0 WatchSource:0}: Error finding container 770ead3d649b1a66a7da222e9a0b3f717a440039b249272c9bec4dcaf79c1bf0: Status 404 returned error can't find the container with id 770ead3d649b1a66a7da222e9a0b3f717a440039b249272c9bec4dcaf79c1bf0 Oct 30 22:39:45.593155 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:45.593116 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" event={"ID":"088f3617-8a8e-490f-9bad-94bba1a95119","Type":"ContainerStarted","Data":"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729"} Oct 30 22:39:45.593563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:45.593184 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" event={"ID":"088f3617-8a8e-490f-9bad-94bba1a95119","Type":"ContainerStarted","Data":"770ead3d649b1a66a7da222e9a0b3f717a440039b249272c9bec4dcaf79c1bf0"} Oct 30 22:39:45.593563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:45.593214 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:39:45.709545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:45.709496 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" podStartSLOduration=1.838915834 podStartE2EDuration="2.709482197s" totalImagesPullingTime="870.566363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:45.707319725 +0000 UTC m=+2306.302408031" watchObservedRunningTime="2026-10-30 22:39:45.709482197 +0000 UTC m=+2306.304570503" Oct 30 22:39:46.110986 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:39:46.110945 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:39:49.415357 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:49.415324 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:39:49.461631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:49.461592 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:39:51.635787 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:39:51.635754 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:40:04.437640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.437582 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" containerID="cri-o://1746ac60b03dfffea6539d493a1c22b5be724cfbc9eddeefe71e7578ef27c455" gracePeriod=45 Oct 30 22:40:04.483376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.483305 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" containerID="cri-o://8491d7d852a1571c7c47d81d6eb7b15c2b014befaee73ef1d300619c23f153b4" gracePeriod=45 Oct 30 22:40:04.498371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.498330 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 30 22:40:04.500700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.500669 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Oct 30 22:40:04.747418 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.747390 4305 generic.go:356] "Generic (PLEG): container finished" podID="a500a836-9f38-40d6-a0da-7ffac0708190" containerID="8491d7d852a1571c7c47d81d6eb7b15c2b014befaee73ef1d300619c23f153b4" exitCode=0 Oct 30 22:40:04.747762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.747735 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" event={"ID":"a500a836-9f38-40d6-a0da-7ffac0708190","Type":"ContainerDied","Data":"8491d7d852a1571c7c47d81d6eb7b15c2b014befaee73ef1d300619c23f153b4"} Oct 30 22:40:04.749622 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.749596 4305 generic.go:356] "Generic (PLEG): container finished" podID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerID="1746ac60b03dfffea6539d493a1c22b5be724cfbc9eddeefe71e7578ef27c455" exitCode=0 Oct 30 22:40:04.749752 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.749680 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" event={"ID":"14f0f1f8-74d1-489c-953d-3ba8e4795494","Type":"ContainerDied","Data":"1746ac60b03dfffea6539d493a1c22b5be724cfbc9eddeefe71e7578ef27c455"} Oct 30 22:40:04.786890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.786864 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:40:04.790506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.790485 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:40:04.971185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.970962 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971060 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.971185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971087 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971185 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971146 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971247 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\" (UniqueName: \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971274 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\" (UniqueName: \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971310 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971363 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\" (UniqueName: \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971427 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971628 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971531 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.971950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971649 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\") pod \"14f0f1f8-74d1-489c-953d-3ba8e4795494\" (UID: \"14f0f1f8-74d1-489c-953d-3ba8e4795494\") " Oct 30 22:40:04.971950 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.971687 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\" (UniqueName: \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\") pod \"a500a836-9f38-40d6-a0da-7ffac0708190\" (UID: \"a500a836-9f38-40d6-a0da-7ffac0708190\") " Oct 30 22:40:04.972140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.972107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.972417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.972397 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:04.973141 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.972783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.973643 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.973452 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.973643 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.973499 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.973643 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.973605 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.974170 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.974002 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.975258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.975237 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "kube-api-access-lts5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:04.976136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.976107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.976136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.976119 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:04.977670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.977635 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s" pod "14f0f1f8-74d1-489c-953d-3ba8e4795494" (UID: "14f0f1f8-74d1-489c-953d-3ba8e4795494"). InnerVolumeSpecName "kube-api-access-djw9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:04.978758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.978723 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:04.978930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:04.978849 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs" pod "a500a836-9f38-40d6-a0da-7ffac0708190" (UID: "a500a836-9f38-40d6-a0da-7ffac0708190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.074993 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075037 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lts5l\" (UniqueName: \"kubernetes.io/projected/a500a836-9f38-40d6-a0da-7ffac0708190-kube-api-access-lts5l\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075052 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075067 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075081 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075099 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a500a836-9f38-40d6-a0da-7ffac0708190-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075112 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f0f1f8-74d1-489c-953d-3ba8e4795494-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075125 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075139 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djw9s\" (UniqueName: \"kubernetes.io/projected/14f0f1f8-74d1-489c-953d-3ba8e4795494-kube-api-access-djw9s\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075151 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f0f1f8-74d1-489c-953d-3ba8e4795494-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.075561 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.075164 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a500a836-9f38-40d6-a0da-7ffac0708190-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:05.758495 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.758407 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" Oct 30 22:40:05.758495 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.758424 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884" event={"ID":"a500a836-9f38-40d6-a0da-7ffac0708190","Type":"ContainerDied","Data":"ae9cb9d7a374761c58677c767a968474c1f6feb23541725fc51e682e91a2d211"} Oct 30 22:40:05.758495 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.758481 4305 scope.go:122] "RemoveContainer" containerID="8491d7d852a1571c7c47d81d6eb7b15c2b014befaee73ef1d300619c23f153b4" Oct 30 22:40:05.760104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.760078 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" event={"ID":"14f0f1f8-74d1-489c-953d-3ba8e4795494","Type":"ContainerDied","Data":"7dbd4c9c7ce2dd5fba07b19cab7c3c4bddd89f400c1e997fbb185de5006804b3"} Oct 30 22:40:05.760245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.760138 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg" Oct 30 22:40:05.768921 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.768901 4305 scope.go:122] "RemoveContainer" containerID="1746ac60b03dfffea6539d493a1c22b5be724cfbc9eddeefe71e7578ef27c455" Oct 30 22:40:05.914215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.914170 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:40:05.959741 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:05.959701 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/distinct-w0-d2d4c3-kserve-84764f7fb5-hb4sg"] Oct 30 22:40:06.020231 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:06.020144 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:40:06.024388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:06.024359 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d2d4c3/premium-w0-d2d4c3-kserve-58d4dc4466-n4884"] Oct 30 22:40:07.568867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.561379 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" path="/var/lib/kubelet/pods/14f0f1f8-74d1-489c-953d-3ba8e4795494/volumes" Oct 30 22:40:07.568867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.562187 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" path="/var/lib/kubelet/pods/a500a836-9f38-40d6-a0da-7ffac0708190/volumes" Oct 30 22:40:07.597853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.597396 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598320 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598345 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598396 4305 cpu_manager.go:414] "containerMap: removing container" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598404 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598534 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a500a836-9f38-40d6-a0da-7ffac0708190" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.598554 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f0f1f8-74d1-489c-953d-3ba8e4795494" containerName="main" Oct 30 22:40:07.610829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.609181 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.642527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.642492 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:40:07.643528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.643491 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" containerID="cri-o://3e2188a7bd5747a607e4a5d29d919520e4d30a61fa03aba79b9d7d391f9ebfce" gracePeriod=30 Oct 30 22:40:07.688559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.688520 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.696626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.696590 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.697098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.697070 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.697940 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.697916 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57jb\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.707695 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.698469 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726540 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.716304 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726646 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726664 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726736 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.711247 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9fce61-openshift-default-dockercfg-bdwvc\"" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726875 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.726924 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" containerID="cri-o://a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621" gracePeriod=30 Oct 30 22:40:07.728308 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.727193 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" containerID="cri-o://ffe34c0180e11bbd225ce590a0e31f6df8237b612d42214147e843548da7e0fc" gracePeriod=30 Oct 30 22:40:07.734625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.731489 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:40:07.827970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.827879 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.827970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.827931 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.827970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.827950 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828014 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828132 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828164 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57jb\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828285 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828341 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828404 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.828865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828843 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.829045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.828870 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.829137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.829104 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.829189 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.829155 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.830516 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.830486 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.831327 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.831310 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.889293 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.889260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:07.889454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:07.889332 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57jb\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\") pod \"e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:08.017963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.017477 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:08.510713 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.510669 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:40:08.549063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.547055 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:08.549063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.547146 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:08.549063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.547183 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:08.789367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:08.789328 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerStarted","Data":"2def4feba663f74a3da58d1ff411628c928978895e4f0e7eeab20b0c9506561e"} Oct 30 22:40:09.828898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:09.828861 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerStarted","Data":"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550"} Oct 30 22:40:10.018519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:10.018481 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:10.024509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:10.024481 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:10.035988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:10.035940 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podStartSLOduration=3.035926727 podStartE2EDuration="3.035926727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:09.988944441 +0000 UTC m=+2330.584032758" watchObservedRunningTime="2026-10-30 22:40:10.035926727 +0000 UTC m=+2330.631015096" Oct 30 22:40:10.835740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:10.835705 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:10.836833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:10.836786 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:40:12.922541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:12.922507 4305 generic.go:356] "Generic (PLEG): container finished" podID="435c8a54-6eb5-4035-99b0-c9e640467500" containerID="ffe34c0180e11bbd225ce590a0e31f6df8237b612d42214147e843548da7e0fc" exitCode=0 Oct 30 22:40:12.923051 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:12.922615 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" event={"ID":"435c8a54-6eb5-4035-99b0-c9e640467500","Type":"ContainerDied","Data":"ffe34c0180e11bbd225ce590a0e31f6df8237b612d42214147e843548da7e0fc"} Oct 30 22:40:12.926160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:12.925229 4305 generic.go:356] "Generic (PLEG): container finished" podID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerID="3e2188a7bd5747a607e4a5d29d919520e4d30a61fa03aba79b9d7d391f9ebfce" exitCode=0 Oct 30 22:40:12.926160 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:12.925706 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" event={"ID":"045ebd6b-0a24-4584-9216-68e4f260adc8","Type":"ContainerDied","Data":"3e2188a7bd5747a607e4a5d29d919520e4d30a61fa03aba79b9d7d391f9ebfce"} Oct 30 22:40:13.388830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.388785 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:40:13.413648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.413621 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:40:13.414139 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414121 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414159 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414212 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414186 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414212 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414241 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414266 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414312 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414346 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414369 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414387 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414420 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414469 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414449 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414504 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414550 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414588 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414606 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.414686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\") pod \"e822d6f0-e475-4032-8e82-2901a5a52768\" (UID: \"e822d6f0-e475-4032-8e82-2901a5a52768\") " Oct 30 22:40:13.414686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.414667 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\") pod \"435c8a54-6eb5-4035-99b0-c9e640467500\" (UID: \"435c8a54-6eb5-4035-99b0-c9e640467500\") " Oct 30 22:40:13.415287 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415233 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.415403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415346 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.415656 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415632 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.415781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415745 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.415901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415865 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.416035 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.415999 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.416270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.416249 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.416587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.416565 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.416833 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.416785 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.418557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.418534 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.418867 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.418776 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "kube-api-access-6h4x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.419342 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.419120 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.419490 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.419256 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.419490 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.419331 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.420278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.420254 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "kube-api-access-64fc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.420354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.420277 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.420966 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.420939 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo" pod "e822d6f0-e475-4032-8e82-2901a5a52768" (UID: "e822d6f0-e475-4032-8e82-2901a5a52768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.421037 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.420968 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy" pod "435c8a54-6eb5-4035-99b0-c9e640467500" (UID: "435c8a54-6eb5-4035-99b0-c9e640467500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.464605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.461564 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:40:13.515962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.515931 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.515990 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516020 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516219 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516173 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516226 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516276 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516328 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516403 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516360 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516440 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.516605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516456 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\") pod \"045ebd6b-0a24-4584-9216-68e4f260adc8\" (UID: \"045ebd6b-0a24-4584-9216-68e4f260adc8\") " Oct 30 22:40:13.516707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516652 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.516985 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.516959 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.517087 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517071 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e822d6f0-e475-4032-8e82-2901a5a52768-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517090 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435c8a54-6eb5-4035-99b0-c9e640467500-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517104 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517119 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517133 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64fc4\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-kube-api-access-64fc4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517149 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517161 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517177 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517190 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517202 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517214 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517227 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4x7\" (UniqueName: \"kubernetes.io/projected/e822d6f0-e475-4032-8e82-2901a5a52768-kube-api-access-6h4x7\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517239 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435c8a54-6eb5-4035-99b0-c9e640467500-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517252 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517268 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e822d6f0-e475-4032-8e82-2901a5a52768-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517287 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045ebd6b-0a24-4584-9216-68e4f260adc8-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517310 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517324 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e822d6f0-e475-4032-8e82-2901a5a52768-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517337 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435c8a54-6eb5-4035-99b0-c9e640467500-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.517423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517345 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435c8a54-6eb5-4035-99b0-c9e640467500-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.518197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517514 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.518197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.517695 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.520292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.520260 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.520541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.520510 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "kube-api-access-jhlmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.521656 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.521631 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.521754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.521743 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo" pod "045ebd6b-0a24-4584-9216-68e4f260adc8" (UID: "045ebd6b-0a24-4584-9216-68e4f260adc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673230 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673269 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhlmd\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-kube-api-access-jhlmd\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673287 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673311 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673325 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045ebd6b-0a24-4584-9216-68e4f260adc8-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673339 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.677127 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.673352 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045ebd6b-0a24-4584-9216-68e4f260adc8-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.943528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.943126 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" Oct 30 22:40:13.943528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.943137 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn" event={"ID":"435c8a54-6eb5-4035-99b0-c9e640467500","Type":"ContainerDied","Data":"4b01128b0e6e0b003d6b76ddf25e5ecc1ea62d6aa6b148c087f58b3b36976d26"} Oct 30 22:40:13.943528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.943194 4305 scope.go:122] "RemoveContainer" containerID="ffe34c0180e11bbd225ce590a0e31f6df8237b612d42214147e843548da7e0fc" Oct 30 22:40:13.973168 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.973142 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" Oct 30 22:40:13.976991 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.973162 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc" event={"ID":"045ebd6b-0a24-4584-9216-68e4f260adc8","Type":"ContainerDied","Data":"80bfba2eec630cd557a2b4f61c25fae00b082d53c46214adb9e8d25e981375dc"} Oct 30 22:40:13.977145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.977032 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" event={"ID":"e822d6f0-e475-4032-8e82-2901a5a52768","Type":"ContainerDied","Data":"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621"} Oct 30 22:40:13.977145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.975078 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" Oct 30 22:40:13.977642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.975030 4305 generic.go:356] "Generic (PLEG): container finished" podID="e822d6f0-e475-4032-8e82-2901a5a52768" containerID="a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621" exitCode=0 Oct 30 22:40:13.977733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.977717 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p" event={"ID":"e822d6f0-e475-4032-8e82-2901a5a52768","Type":"ContainerDied","Data":"b7c2ef3e049594d3937cc22370c0d789c43e5f1c00ed21ebe67bc66bcd6e42c2"} Oct 30 22:40:13.985835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.985817 4305 scope.go:122] "RemoveContainer" containerID="3e2188a7bd5747a607e4a5d29d919520e4d30a61fa03aba79b9d7d391f9ebfce" Oct 30 22:40:13.995149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:13.995122 4305 scope.go:122] "RemoveContainer" containerID="a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621" Oct 30 22:40:14.005075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.005054 4305 scope.go:122] "RemoveContainer" containerID="a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621" Oct 30 22:40:14.005428 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:14.005407 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621\": container with ID starting with a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621 not found: ID does not exist" containerID="a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621" Oct 30 22:40:14.005504 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.005443 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621"} err="failed to get container status \"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621\": rpc error: code = NotFound desc = could not find container \"a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621\": container with ID starting with a5247c85b4500b92f512fbbe1af64364e06aecf60cebc8d96f7b2e90cc773621 not found: ID does not exist" Oct 30 22:40:14.099375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.099328 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:40:14.103351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.103319 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-z7r5p"] Oct 30 22:40:14.205852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.205745 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:40:14.213111 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.213078 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-ncnnc"] Oct 30 22:40:14.315884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.315852 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:40:14.322113 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:14.322081 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-np6gn"] Oct 30 22:40:15.520041 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:15.520000 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" path="/var/lib/kubelet/pods/045ebd6b-0a24-4584-9216-68e4f260adc8/volumes" Oct 30 22:40:15.520492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:15.520478 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" path="/var/lib/kubelet/pods/435c8a54-6eb5-4035-99b0-c9e640467500/volumes" Oct 30 22:40:15.520935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:15.520920 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" path="/var/lib/kubelet/pods/e822d6f0-e475-4032-8e82-2901a5a52768/volumes" Oct 30 22:40:15.901012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:15.900919 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:40:15.901201 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:15.901000 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:40:16.113601 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:16.113562 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:40:19.017875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.017836 4305 generic.go:356] "Generic (PLEG): container finished" podID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerID="a35cf403f47bb00621799616a80b187dd1b3275a630b23c6775a2a170e061978" exitCode=0 Oct 30 22:40:19.018329 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.017896 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerDied","Data":"a35cf403f47bb00621799616a80b187dd1b3275a630b23c6775a2a170e061978"} Oct 30 22:40:19.018499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.018479 4305 scope.go:122] "RemoveContainer" containerID="a35cf403f47bb00621799616a80b187dd1b3275a630b23c6775a2a170e061978" Oct 30 22:40:19.020239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.020213 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:19.020334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.020283 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:19.020334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:19.020316 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:20.026499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:20.026464 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerStarted","Data":"78dfb2e1111b58d26d99357b862751f16d0cd2af21f2c1f0069e21636e96ba34"} Oct 30 22:40:20.071836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:20.068229 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:40:20.071836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:20.068360 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:20.071836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:20.068403 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:21.071818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:21.071350 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:21.071818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:21.071450 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:22.064832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.064230 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:22.064832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.064300 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:22.316559 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.316484 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:40:22.317012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.316703 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" containerID="cri-o://e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591" gracePeriod=30 Oct 30 22:40:22.323270 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.323243 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:40:22.323454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:22.323433 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" containerID="cri-o://56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549" gracePeriod=30 Oct 30 22:40:23.063525 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:23.063104 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:23.063525 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:23.063164 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:24.068823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:24.064141 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:24.068823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:24.064214 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:25.052126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:25.052090 4305 patch_prober.go:29] interesting pod/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:40:25.052354 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:25.052145 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 22:40:26.057136 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:26.057103 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:40:26.057623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:26.057398 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:40:26.058214 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:26.058197 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:40:27.610541 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.610514 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:40:27.625140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.625118 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:40:27.653448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653422 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.653576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653482 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.653576 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653528 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.653675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653575 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.653675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653613 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.653675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653660 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.653840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653694 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.653840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653692 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.653840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653734 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.653840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653771 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.653840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653873 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653960 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653998 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.653997 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654027 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654060 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654082 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:27.654108 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654087 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.654575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654134 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.654575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654160 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.654575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654223 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\") pod \"892db7da-515e-4c64-9ede-c55394a87f03\" (UID: \"892db7da-515e-4c64-9ede-c55394a87f03\") " Oct 30 22:40:27.654575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654268 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\") pod \"105dc63b-a219-4f9f-840f-a97f1247be5b\" (UID: \"105dc63b-a219-4f9f-840f-a97f1247be5b\") " Oct 30 22:40:27.654575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654535 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654848 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654877 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.654894 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/105dc63b-a219-4f9f-840f-a97f1247be5b-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655182 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655196 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655303 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655948 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655977 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.655993 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.656064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.656008 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.657081 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657056 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.657341 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657305 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:27.657428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657339 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.657476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657451 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.657608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657578 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:27.657772 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657749 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.657856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.657833 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4" pod "105dc63b-a219-4f9f-840f-a97f1247be5b" (UID: "105dc63b-a219-4f9f-840f-a97f1247be5b"). InnerVolumeSpecName "kube-api-access-m9gz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.658353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.658334 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm" pod "892db7da-515e-4c64-9ede-c55394a87f03" (UID: "892db7da-515e-4c64-9ede-c55394a87f03"). InnerVolumeSpecName "kube-api-access-b87rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.757345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757311 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892db7da-515e-4c64-9ede-c55394a87f03-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757343 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757353 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757364 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757373 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87rm\" (UniqueName: \"kubernetes.io/projected/892db7da-515e-4c64-9ede-c55394a87f03-kube-api-access-b87rm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757385 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757393 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892db7da-515e-4c64-9ede-c55394a87f03-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757401 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757409 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/105dc63b-a219-4f9f-840f-a97f1247be5b-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757418 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892db7da-515e-4c64-9ede-c55394a87f03-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.757589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:27.757428 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gz4\" (UniqueName: \"kubernetes.io/projected/105dc63b-a219-4f9f-840f-a97f1247be5b-kube-api-access-m9gz4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.099666 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.099623 4305 generic.go:356] "Generic (PLEG): container finished" podID="892db7da-515e-4c64-9ede-c55394a87f03" containerID="e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591" exitCode=0 Oct 30 22:40:28.099894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.099674 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" Oct 30 22:40:28.099894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.099713 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" event={"ID":"892db7da-515e-4c64-9ede-c55394a87f03","Type":"ContainerDied","Data":"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591"} Oct 30 22:40:28.099894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.099775 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz" event={"ID":"892db7da-515e-4c64-9ede-c55394a87f03","Type":"ContainerDied","Data":"7e3f2b4f5f1d563dd1f64b5f81752b5bf997d2e1d231804b28f3faae086d7595"} Oct 30 22:40:28.099894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.099829 4305 scope.go:122] "RemoveContainer" containerID="e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591" Oct 30 22:40:28.101616 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.101590 4305 generic.go:356] "Generic (PLEG): container finished" podID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerID="56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549" exitCode=0 Oct 30 22:40:28.101759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.101631 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerDied","Data":"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549"} Oct 30 22:40:28.101759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.101662 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" event={"ID":"105dc63b-a219-4f9f-840f-a97f1247be5b","Type":"ContainerDied","Data":"01c7f0759403ec22d4708af132d8e3bba6982a10d2684b3f0f21889987f2a2c1"} Oct 30 22:40:28.101759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.101714 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b" Oct 30 22:40:28.111689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.111672 4305 scope.go:122] "RemoveContainer" containerID="e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591" Oct 30 22:40:28.112012 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:28.111945 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591\": container with ID starting with e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591 not found: ID does not exist" containerID="e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591" Oct 30 22:40:28.112012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.111979 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591"} err="failed to get container status \"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591\": rpc error: code = NotFound desc = could not find container \"e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591\": container with ID starting with e5fa9e1460bcc4c23c0eebb80251ccf703805e8d7e0c5002a9e0bb672958a591 not found: ID does not exist" Oct 30 22:40:28.112012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.111997 4305 scope.go:122] "RemoveContainer" containerID="56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549" Oct 30 22:40:28.120621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.120605 4305 scope.go:122] "RemoveContainer" containerID="70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14" Oct 30 22:40:28.129088 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.129071 4305 scope.go:122] "RemoveContainer" containerID="56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549" Oct 30 22:40:28.129375 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:28.129356 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549\": container with ID starting with 56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549 not found: ID does not exist" containerID="56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549" Oct 30 22:40:28.129430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.129382 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549"} err="failed to get container status \"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549\": rpc error: code = NotFound desc = could not find container \"56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549\": container with ID starting with 56b3f4b2af2e08f4ee25886197b63b9644792f53fbdc6463594017c6fa0a4549 not found: ID does not exist" Oct 30 22:40:28.129430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.129401 4305 scope.go:122] "RemoveContainer" containerID="70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14" Oct 30 22:40:28.129633 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:28.129616 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14\": container with ID starting with 70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14 not found: ID does not exist" containerID="70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14" Oct 30 22:40:28.129671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.129643 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14"} err="failed to get container status \"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14\": rpc error: code = NotFound desc = could not find container \"70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14\": container with ID starting with 70718ad90f5807183f611fc4d0047c32461228eb06e95494ccf4a59f3fa39b14 not found: ID does not exist" Oct 30 22:40:28.256044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.256003 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:40:28.301982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.301947 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-779lz"] Oct 30 22:40:28.404601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.404564 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:40:28.411121 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:28.411092 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-hnp7b"] Oct 30 22:40:29.528078 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:29.528043 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" path="/var/lib/kubelet/pods/105dc63b-a219-4f9f-840f-a97f1247be5b/volumes" Oct 30 22:40:29.528563 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:29.528548 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892db7da-515e-4c64-9ede-c55394a87f03" path="/var/lib/kubelet/pods/892db7da-515e-4c64-9ede-c55394a87f03/volumes" Oct 30 22:40:42.236500 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.236464 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:40:42.236996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.236682 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" podUID="838c9a5b-657f-4270-86ad-1b643dc837a4" containerName="maas-api" containerID="cri-o://2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc" gracePeriod=30 Oct 30 22:40:42.559309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.559280 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:40:42.617818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.614416 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\" (UniqueName: \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\") pod \"838c9a5b-657f-4270-86ad-1b643dc837a4\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " Oct 30 22:40:42.617818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.614481 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\") pod \"838c9a5b-657f-4270-86ad-1b643dc837a4\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " Oct 30 22:40:42.617818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.614532 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") pod \"838c9a5b-657f-4270-86ad-1b643dc837a4\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " Oct 30 22:40:42.617818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.614684 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\") pod \"838c9a5b-657f-4270-86ad-1b643dc837a4\" (UID: \"838c9a5b-657f-4270-86ad-1b643dc837a4\") " Oct 30 22:40:42.637542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.634026 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca" pod "838c9a5b-657f-4270-86ad-1b643dc837a4" (UID: "838c9a5b-657f-4270-86ad-1b643dc837a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:42.680013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.679948 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls" pod "838c9a5b-657f-4270-86ad-1b643dc837a4" (UID: "838c9a5b-657f-4270-86ad-1b643dc837a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:42.685120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.685067 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr" pod "838c9a5b-657f-4270-86ad-1b643dc837a4" (UID: "838c9a5b-657f-4270-86ad-1b643dc837a4"). InnerVolumeSpecName "kube-api-access-pgxgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:42.696001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.692404 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs" pod "838c9a5b-657f-4270-86ad-1b643dc837a4" (UID: "838c9a5b-657f-4270-86ad-1b643dc837a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:42.728286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.727533 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/838c9a5b-657f-4270-86ad-1b643dc837a4-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:42.728286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.727571 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxgr\" (UniqueName: \"kubernetes.io/projected/838c9a5b-657f-4270-86ad-1b643dc837a4-kube-api-access-pgxgr\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:42.728286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.727588 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:42.728286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:42.727603 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/838c9a5b-657f-4270-86ad-1b643dc837a4-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:43.263542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.263508 4305 generic.go:356] "Generic (PLEG): container finished" podID="838c9a5b-657f-4270-86ad-1b643dc837a4" containerID="2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc" exitCode=0 Oct 30 22:40:43.264104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.263669 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" event={"ID":"838c9a5b-657f-4270-86ad-1b643dc837a4","Type":"ContainerDied","Data":"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc"} Oct 30 22:40:43.264104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.263709 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" event={"ID":"838c9a5b-657f-4270-86ad-1b643dc837a4","Type":"ContainerDied","Data":"fb26de5e924be5c13669d9a479730550d9b05cdc1fb16edcc787aef3b64454c6"} Oct 30 22:40:43.264104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.263730 4305 scope.go:122] "RemoveContainer" containerID="2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc" Oct 30 22:40:43.264104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.263890 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n" Oct 30 22:40:43.276091 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.276067 4305 scope.go:122] "RemoveContainer" containerID="2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc" Oct 30 22:40:43.276431 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:43.276407 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc\": container with ID starting with 2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc not found: ID does not exist" containerID="2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc" Oct 30 22:40:43.276514 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.276446 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc"} err="failed to get container status \"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc\": rpc error: code = NotFound desc = could not find container \"2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc\": container with ID starting with 2cfad6314fba612f7700a7c1ee7e2bb46acaea636c55ec2e69ace6d9ca5b2acc not found: ID does not exist" Oct 30 22:40:43.426201 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.426167 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:40:43.471510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.471475 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-af30a9-5f4bf6f48b-tpk6n"] Oct 30 22:40:43.519466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:43.519390 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838c9a5b-657f-4270-86ad-1b643dc837a4" path="/var/lib/kubelet/pods/838c9a5b-657f-4270-86ad-1b643dc837a4/volumes" Oct 30 22:40:46.109536 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:40:46.109502 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:40:51.257364 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:51.257326 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:40:51.257850 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:51.257523 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" containerID="cri-o://31aefe07c5a806d8ad49fc2b790ef796c9e6cc5a4642203f4bcd179759c8a59f" gracePeriod=30 Oct 30 22:40:53.132712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:53.132672 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:40:53.133102 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:53.132745 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:40:56.381528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.381496 4305 generic.go:356] "Generic (PLEG): container finished" podID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerID="31aefe07c5a806d8ad49fc2b790ef796c9e6cc5a4642203f4bcd179759c8a59f" exitCode=0 Oct 30 22:40:56.381866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.381571 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" event={"ID":"b5deb917-8d4c-4fbe-b863-40314d06facd","Type":"ContainerDied","Data":"31aefe07c5a806d8ad49fc2b790ef796c9e6cc5a4642203f4bcd179759c8a59f"} Oct 30 22:40:56.515457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.515433 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:40:56.544169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544112 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544169 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544157 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544184 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544295 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544345 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544344 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544405 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544417 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.544503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544431 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.544503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544443 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544524 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544534 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.544715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544566 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\") pod \"b5deb917-8d4c-4fbe-b863-40314d06facd\" (UID: \"b5deb917-8d4c-4fbe-b863-40314d06facd\") " Oct 30 22:40:56.544979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.544979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.544858 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:56.545386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.545265 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.545386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.545301 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.545386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.545320 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.545386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.545335 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5deb917-8d4c-4fbe-b863-40314d06facd-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.545386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.545351 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.546607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.546572 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.546705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.546631 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "kube-api-access-wn229". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:56.546748 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.546723 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:56.546789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.546758 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo" pod "b5deb917-8d4c-4fbe-b863-40314d06facd" (UID: "b5deb917-8d4c-4fbe-b863-40314d06facd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:56.646447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.646419 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.646447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.646443 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.646447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.646453 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5deb917-8d4c-4fbe-b863-40314d06facd-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.646654 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:56.646462 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn229\" (UniqueName: \"kubernetes.io/projected/b5deb917-8d4c-4fbe-b863-40314d06facd-kube-api-access-wn229\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:57.268322 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268289 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:40:57.268898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268869 4305 cpu_manager.go:414] "containerMap: removing container" podUID="838c9a5b-657f-4270-86ad-1b643dc837a4" containerName="maas-api" Oct 30 22:40:57.268962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268921 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838c9a5b-657f-4270-86ad-1b643dc837a4" containerName="maas-api" Oct 30 22:40:57.268962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268937 4305 cpu_manager.go:414] "containerMap: removing container" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.268962 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268944 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.268989 4305 cpu_manager.go:414] "containerMap: removing container" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269008 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269031 4305 cpu_manager.go:414] "containerMap: removing container" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269036 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269083 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269091 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" Oct 30 22:40:57.269104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269101 4305 cpu_manager.go:414] "containerMap: removing container" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269108 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269118 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269123 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269155 4305 cpu_manager.go:414] "containerMap: removing container" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269160 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269275 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="045ebd6b-0a24-4584-9216-68e4f260adc8" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269286 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="838c9a5b-657f-4270-86ad-1b643dc837a4" containerName="maas-api" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269299 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269307 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="105dc63b-a219-4f9f-840f-a97f1247be5b" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269314 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="892db7da-515e-4c64-9ede-c55394a87f03" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269321 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269327 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="435c8a54-6eb5-4035-99b0-c9e640467500" containerName="istio-proxy" Oct 30 22:40:57.269340 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.269333 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e822d6f0-e475-4032-8e82-2901a5a52768" containerName="istio-proxy" Oct 30 22:40:57.271877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.271855 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.314816 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.314771 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:40:57.318241 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.318218 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.354571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354542 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.354713 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354588 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.354713 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354615 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.354869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354841 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.354935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354896 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355005 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.354974 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355025 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355123 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhlqw\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355166 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355333 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355312 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwd6x\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355347 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355393 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355479 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355513 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355541 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.355763 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355586 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.355763 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.355734 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.368257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.368230 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:40:57.372731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.372706 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:40:57.392082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.392059 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" Oct 30 22:40:57.392451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.392102 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh" event={"ID":"b5deb917-8d4c-4fbe-b863-40314d06facd","Type":"ContainerDied","Data":"04639d3156ef75f3b85635bcacaf0bec2ebb28f8ab7905451ce67926e1063e7e"} Oct 30 22:40:57.392451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.392168 4305 scope.go:122] "RemoveContainer" containerID="31aefe07c5a806d8ad49fc2b790ef796c9e6cc5a4642203f4bcd179759c8a59f" Oct 30 22:40:57.456626 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456598 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.456733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456641 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.456733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456672 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.456733 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456707 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.456922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456738 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.456922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456763 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhlqw\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.456922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456835 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.456922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456906 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwd6x\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456931 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.456974 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457000 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457038 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457057 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457070 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457092 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457118 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457139 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457162 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457250 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457316 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457326 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457377 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457592 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457478 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.457929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457605 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.457929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.457770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.458036 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.458015 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.458257 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.458234 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.458437 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.458411 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.459589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.459560 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.459857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.459829 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.460135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.460112 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.460307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.460287 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.514778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.514752 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.515048 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.515024 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.515151 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.515127 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhlqw\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.515278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.515260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwd6x\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:57.527199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.527131 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:40:57.529404 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.529375 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-rgzlh"] Oct 30 22:40:57.586286 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.586265 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:57.638892 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:57.638615 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:58.011507 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.011476 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:40:58.016672 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:40:58.016643 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0bd1c2_3271_43d8_a9e8_d506784e77e1.slice/crio-851e788831acf668c0c6c2309372c16d324f214a7c79c986976f4d3f50b282be WatchSource:0}: Error finding container 851e788831acf668c0c6c2309372c16d324f214a7c79c986976f4d3f50b282be: Status 404 returned error can't find the container with id 851e788831acf668c0c6c2309372c16d324f214a7c79c986976f4d3f50b282be Oct 30 22:40:58.019147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.019115 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.019211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.019185 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.019254 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.019212 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.067918 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.064558 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:40:58.069654 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:40:58.069630 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0915ef56_8ab1_4e37_a043_13f267ad87c4.slice/crio-dbbd7d869400ac3697074b8b696805289b9de1fdc6e085cfbffc25a14f8cc115 WatchSource:0}: Error finding container dbbd7d869400ac3697074b8b696805289b9de1fdc6e085cfbffc25a14f8cc115: Status 404 returned error can't find the container with id dbbd7d869400ac3697074b8b696805289b9de1fdc6e085cfbffc25a14f8cc115 Oct 30 22:40:58.072611 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.072580 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.072693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.072643 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.072693 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.072668 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:40:58.453520 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.453480 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" event={"ID":"fd0bd1c2-3271-43d8-a9e8-d506784e77e1","Type":"ContainerStarted","Data":"c706a82821fd7415e79b835d639b40bb8489671ccabb5a172c4a69f71439e97a"} Oct 30 22:40:58.454064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.453549 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" event={"ID":"fd0bd1c2-3271-43d8-a9e8-d506784e77e1","Type":"ContainerStarted","Data":"851e788831acf668c0c6c2309372c16d324f214a7c79c986976f4d3f50b282be"} Oct 30 22:40:58.456904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.456872 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" event={"ID":"0915ef56-8ab1-4e37-a043-13f267ad87c4","Type":"ContainerStarted","Data":"f8e8883848eb3a2e47731010084a2755eeb56070ad4965fced72ae476d3160aa"} Oct 30 22:40:58.457063 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.456924 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" event={"ID":"0915ef56-8ab1-4e37-a043-13f267ad87c4","Type":"ContainerStarted","Data":"dbbd7d869400ac3697074b8b696805289b9de1fdc6e085cfbffc25a14f8cc115"} Oct 30 22:40:58.570548 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.570489 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podStartSLOduration=1.570473888 podStartE2EDuration="1.570473888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:58.568034745 +0000 UTC m=+2379.163123063" watchObservedRunningTime="2026-10-30 22:40:58.570473888 +0000 UTC m=+2379.165562194" Oct 30 22:40:58.586963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.586932 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:40:58.639485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.639452 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:40:58.691028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:58.690961 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podStartSLOduration=1.6909391299999998 podStartE2EDuration="1.69093913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:58.686325138 +0000 UTC m=+2379.281413467" watchObservedRunningTime="2026-10-30 22:40:58.69093913 +0000 UTC m=+2379.286027451" Oct 30 22:40:59.519150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:59.519118 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5deb917-8d4c-4fbe-b863-40314d06facd" path="/var/lib/kubelet/pods/b5deb917-8d4c-4fbe-b863-40314d06facd/volumes" Oct 30 22:40:59.601822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:59.601045 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:40:59.601822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:59.601108 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:40:59.640860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:59.640829 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:40:59.640993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:40:59.640876 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:00.595825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:00.595232 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:41:00.595825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:00.595312 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:41:00.640045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:00.640008 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:41:00.640209 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:00.640056 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:01.591821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:01.591120 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:41:01.591821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:01.591193 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:41:01.639535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:01.639504 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:41:01.639974 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:01.639564 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:02.588822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:02.588235 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:41:02.588822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:02.588302 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:41:02.639510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:02.639478 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:41:02.639683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:02.639522 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:03.588823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:03.588211 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:41:03.588823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:03.588271 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:41:03.639304 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:03.639267 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:41:03.639472 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:03.639332 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:04.587184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:04.587142 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:41:04.587602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:04.587212 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:41:04.639684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:04.639648 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:41:04.639860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:04.639712 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 22:41:05.591456 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.591424 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:41:05.591938 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.591781 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:41:05.592483 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.592461 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:41:05.643441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.643420 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:41:05.643825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.643776 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:41:05.644450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.644435 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:41:05.913671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.913593 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:41:05.917536 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.917512 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:05.990683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:05.990643 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:41:06.030602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.030563 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.030792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.030664 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8w6\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.030792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.030740 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.030953 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.030923 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.031010 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.030995 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.031109 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.031090 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.031165 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.031122 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.031227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.031168 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.031289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.031236 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132498 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132460 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132515 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132565 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132583 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132634 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132697 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132784 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132843 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8w6\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132924 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.132967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.132946 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.133265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.133010 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.133265 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.133249 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.133406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.133388 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.133527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.133501 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.135097 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.135064 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.135324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.135309 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.190421 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.190394 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.190871 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.190847 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8w6\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fj2bq\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.233647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.233619 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:06.661013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.660934 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:41:06.665878 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:41:06.665850 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a306129_d341_4737_b068_dd0b3c46aa1c.slice/crio-ed47f943010099a03aec5740a719a13c6e1d7d94a87b06b4c480ddeeb626d9bf WatchSource:0}: Error finding container ed47f943010099a03aec5740a719a13c6e1d7d94a87b06b4c480ddeeb626d9bf: Status 404 returned error can't find the container with id ed47f943010099a03aec5740a719a13c6e1d7d94a87b06b4c480ddeeb626d9bf Oct 30 22:41:06.668104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.668073 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:06.668191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.668136 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:06.668191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:06.668161 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:07.577426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:07.577382 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" event={"ID":"7a306129-d341-4737-b068-dd0b3c46aa1c","Type":"ContainerStarted","Data":"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90"} Oct 30 22:41:07.577640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:07.577443 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" event={"ID":"7a306129-d341-4737-b068-dd0b3c46aa1c","Type":"ContainerStarted","Data":"ed47f943010099a03aec5740a719a13c6e1d7d94a87b06b4c480ddeeb626d9bf"} Oct 30 22:41:07.697961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:07.697895 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podStartSLOduration=2.6978760360000003 podStartE2EDuration="2.697876036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:07.693648465 +0000 UTC m=+2388.288736786" watchObservedRunningTime="2026-10-30 22:41:07.697876036 +0000 UTC m=+2388.292964345" Oct 30 22:41:08.234031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:08.233994 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:08.235754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:08.235731 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:08.235828 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:08.235780 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:09.234317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:09.234280 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:09.234720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:09.234349 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:10.234729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:10.234694 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:10.235137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:10.234751 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:11.234332 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:11.234295 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:11.234520 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:11.234351 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:12.238660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:12.238314 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:12.238660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:12.238389 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:13.234269 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:13.234230 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:13.234454 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:13.234300 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:14.234213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:14.234182 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fj2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:41:14.234637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:14.234241 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 22:41:15.305674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:15.305642 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:15.306614 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:15.306581 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:15.308112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:15.308091 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:41:16.116878 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:41:16.116836 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:41:16.256663 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.256627 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:41:16.256874 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.256850 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" containerName="maas-api" containerID="cri-o://f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076" gracePeriod=30 Oct 30 22:41:16.590496 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.590471 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.630762 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\") pod \"ca71e3e9-ada3-4096-a730-504f508521cf\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.631257 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca" pod "ca71e3e9-ada3-4096-a730-504f508521cf" (UID: "ca71e3e9-ada3-4096-a730-504f508521cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.631478 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") pod \"ca71e3e9-ada3-4096-a730-504f508521cf\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.631520 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\" (UniqueName: \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\") pod \"ca71e3e9-ada3-4096-a730-504f508521cf\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.631603 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\") pod \"ca71e3e9-ada3-4096-a730-504f508521cf\" (UID: \"ca71e3e9-ada3-4096-a730-504f508521cf\") " Oct 30 22:41:16.632889 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.632281 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca71e3e9-ada3-4096-a730-504f508521cf-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:16.634696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.634670 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj" pod "ca71e3e9-ada3-4096-a730-504f508521cf" (UID: "ca71e3e9-ada3-4096-a730-504f508521cf"). InnerVolumeSpecName "kube-api-access-fpczj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:16.637634 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.637528 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls" pod "ca71e3e9-ada3-4096-a730-504f508521cf" (UID: "ca71e3e9-ada3-4096-a730-504f508521cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:16.640897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.640859 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs" pod "ca71e3e9-ada3-4096-a730-504f508521cf" (UID: "ca71e3e9-ada3-4096-a730-504f508521cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:16.645203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.642701 4305 generic.go:356] "Generic (PLEG): container finished" podID="ca71e3e9-ada3-4096-a730-504f508521cf" containerID="f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076" exitCode=0 Oct 30 22:41:16.645203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.642761 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" Oct 30 22:41:16.645203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.642784 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" event={"ID":"ca71e3e9-ada3-4096-a730-504f508521cf","Type":"ContainerDied","Data":"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076"} Oct 30 22:41:16.645203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.642886 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" event={"ID":"ca71e3e9-ada3-4096-a730-504f508521cf","Type":"ContainerDied","Data":"b7d4467031dfef75b6a500e2c38e36e16a95e3e1d2a1093b1653de975ab6f5a2"} Oct 30 22:41:16.645203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.642904 4305 scope.go:122] "RemoveContainer" containerID="f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076" Oct 30 22:41:16.666639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.666621 4305 scope.go:122] "RemoveContainer" containerID="f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076" Oct 30 22:41:16.666952 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:41:16.666923 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076\": container with ID starting with f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076 not found: ID does not exist" containerID="f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076" Oct 30 22:41:16.667002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.666962 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076"} err="failed to get container status \"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076\": rpc error: code = NotFound desc = could not find container \"f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076\": container with ID starting with f952215e6155529f19aa8be4ba83b76635a02b9323503bebf3d1d6866fd3b076 not found: ID does not exist" Oct 30 22:41:16.734199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.734168 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:16.734199 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.734202 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpczj\" (UniqueName: \"kubernetes.io/projected/ca71e3e9-ada3-4096-a730-504f508521cf-kube-api-access-fpczj\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:16.734370 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.734215 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca71e3e9-ada3-4096-a730-504f508521cf-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:16.808441 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.808406 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:41:16.812538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:16.812514 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb"] Oct 30 22:41:17.524125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:17.524082 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" path="/var/lib/kubelet/pods/ca71e3e9-ada3-4096-a730-504f508521cf/volumes" Oct 30 22:41:19.570972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:19.570938 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d2d4c3-6db7cc8fb9-vprbb" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": context deadline exceeded" Oct 30 22:41:19.855381 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:19.855307 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:41:19.858571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:19.858545 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:41:25.371225 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.371187 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:41:25.371642 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.371444 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" containerID="cri-o://c706a82821fd7415e79b835d639b40bb8489671ccabb5a172c4a69f71439e97a" gracePeriod=30 Oct 30 22:41:25.383601 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.383569 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:41:25.383792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.383769 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" containerID="cri-o://1974ffeec86a603f5743bebef08047c013ebeb270f7f21c9fd579d45287ee8a3" gracePeriod=30 Oct 30 22:41:25.409256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.409226 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:41:25.409474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:25.409450 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" containerID="cri-o://f8e8883848eb3a2e47731010084a2755eeb56070ad4965fced72ae476d3160aa" gracePeriod=30 Oct 30 22:41:27.602470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:27.602434 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:41:27.602965 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:27.602624 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" containerID="cri-o://ff2db3badaaad33ce1b102cc6473b51bdcca42488ffd5425215d47e18c615e52" gracePeriod=30 Oct 30 22:41:27.953845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:27.950878 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:27.953845 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:27.950941 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:30.737300 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.737263 4305 generic.go:356] "Generic (PLEG): container finished" podID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerID="c706a82821fd7415e79b835d639b40bb8489671ccabb5a172c4a69f71439e97a" exitCode=0 Oct 30 22:41:30.737689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.737394 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" event={"ID":"fd0bd1c2-3271-43d8-a9e8-d506784e77e1","Type":"ContainerDied","Data":"c706a82821fd7415e79b835d639b40bb8489671ccabb5a172c4a69f71439e97a"} Oct 30 22:41:30.740107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.740077 4305 generic.go:356] "Generic (PLEG): container finished" podID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerID="1974ffeec86a603f5743bebef08047c013ebeb270f7f21c9fd579d45287ee8a3" exitCode=0 Oct 30 22:41:30.740210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.740157 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" event={"ID":"c9896e29-0f85-49f1-94d9-7597e266f71c","Type":"ContainerDied","Data":"1974ffeec86a603f5743bebef08047c013ebeb270f7f21c9fd579d45287ee8a3"} Oct 30 22:41:30.743407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.743029 4305 generic.go:356] "Generic (PLEG): container finished" podID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerID="f8e8883848eb3a2e47731010084a2755eeb56070ad4965fced72ae476d3160aa" exitCode=0 Oct 30 22:41:30.743407 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.743065 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" event={"ID":"0915ef56-8ab1-4e37-a043-13f267ad87c4","Type":"ContainerDied","Data":"f8e8883848eb3a2e47731010084a2755eeb56070ad4965fced72ae476d3160aa"} Oct 30 22:41:30.793489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.793263 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:41:30.805945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.805202 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:41:30.842003 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.841971 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:41:30.901200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901109 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901148 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901187 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901220 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901305 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901342 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901384 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901412 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901439 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901457 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901478 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901518 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901546 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901552 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901566 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901586 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901619 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901670 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901720 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901765 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901750 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901847 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901874 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901886 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901901 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.901948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901934 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\") pod \"c9896e29-0f85-49f1-94d9-7597e266f71c\" (UID: \"c9896e29-0f85-49f1-94d9-7597e266f71c\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.901999 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902020 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902033 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902044 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902063 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902109 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902142 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\") pod \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\" (UID: \"fd0bd1c2-3271-43d8-a9e8-d506784e77e1\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902173 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902211 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902212 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902253 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\") pod \"0915ef56-8ab1-4e37-a043-13f267ad87c4\" (UID: \"0915ef56-8ab1-4e37-a043-13f267ad87c4\") " Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902279 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.902603 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902371 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902851 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902873 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9896e29-0f85-49f1-94d9-7597e266f71c-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902888 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902902 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902918 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902934 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902948 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902963 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902978 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.902993 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.903009 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.903023 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.903367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.903341 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:30.904710 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.904680 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.904839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.904739 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:30.904839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.904757 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:30.905064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.905036 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.905765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.905488 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "kube-api-access-cwd6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.905939 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.905915 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.906309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.906285 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:30.906417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.906374 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.906761 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.906736 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.907278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.907153 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "kube-api-access-fqntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.907278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.907245 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.907754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.907730 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw" pod "fd0bd1c2-3271-43d8-a9e8-d506784e77e1" (UID: "fd0bd1c2-3271-43d8-a9e8-d506784e77e1"). InnerVolumeSpecName "kube-api-access-nhlqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.907984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.907954 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo" pod "0915ef56-8ab1-4e37-a043-13f267ad87c4" (UID: "0915ef56-8ab1-4e37-a043-13f267ad87c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:30.908375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:30.908355 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token" pod "c9896e29-0f85-49f1-94d9-7597e266f71c" (UID: "c9896e29-0f85-49f1-94d9-7597e266f71c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:31.004313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004276 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004317 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqntg\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-kube-api-access-fqntg\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004334 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004348 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004362 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004376 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004392 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhlqw\" (UniqueName: \"kubernetes.io/projected/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-kube-api-access-nhlqw\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004407 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004421 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004434 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0915ef56-8ab1-4e37-a043-13f267ad87c4-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004448 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd0bd1c2-3271-43d8-a9e8-d506784e77e1-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004467 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004464 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004478 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwd6x\" (UniqueName: \"kubernetes.io/projected/0915ef56-8ab1-4e37-a043-13f267ad87c4-kube-api-access-cwd6x\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004492 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.004853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.004505 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9896e29-0f85-49f1-94d9-7597e266f71c-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.775005 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.774965 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" event={"ID":"c9896e29-0f85-49f1-94d9-7597e266f71c","Type":"ContainerDied","Data":"a9c67d418760b46745007d3fdf95d0655f7e01b9de915b62daecb7466777ac41"} Oct 30 22:41:31.775005 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.774988 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj" Oct 30 22:41:31.775510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.775036 4305 scope.go:122] "RemoveContainer" containerID="1974ffeec86a603f5743bebef08047c013ebeb270f7f21c9fd579d45287ee8a3" Oct 30 22:41:31.776975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.776954 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" Oct 30 22:41:31.776975 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.776960 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s" event={"ID":"0915ef56-8ab1-4e37-a043-13f267ad87c4","Type":"ContainerDied","Data":"dbbd7d869400ac3697074b8b696805289b9de1fdc6e085cfbffc25a14f8cc115"} Oct 30 22:41:31.778646 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.778622 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" event={"ID":"fd0bd1c2-3271-43d8-a9e8-d506784e77e1","Type":"ContainerDied","Data":"851e788831acf668c0c6c2309372c16d324f214a7c79c986976f4d3f50b282be"} Oct 30 22:41:31.778766 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.778677 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5" Oct 30 22:41:31.785899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.785877 4305 scope.go:122] "RemoveContainer" containerID="f8e8883848eb3a2e47731010084a2755eeb56070ad4965fced72ae476d3160aa" Oct 30 22:41:31.795368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.795349 4305 scope.go:122] "RemoveContainer" containerID="c706a82821fd7415e79b835d639b40bb8489671ccabb5a172c4a69f71439e97a" Oct 30 22:41:31.941736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.941702 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:41:31.987474 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:31.987439 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-b494s"] Oct 30 22:41:32.072989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.072885 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:41:32.077386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.077350 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-qmwq5"] Oct 30 22:41:32.175128 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.175088 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:41:32.179568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.179538 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-mqrxj"] Oct 30 22:41:32.786613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.786581 4305 generic.go:356] "Generic (PLEG): container finished" podID="09df02e4-d0fd-4186-b605-02932d5739d1" containerID="ff2db3badaaad33ce1b102cc6473b51bdcca42488ffd5425215d47e18c615e52" exitCode=0 Oct 30 22:41:32.787079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.786711 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" event={"ID":"09df02e4-d0fd-4186-b605-02932d5739d1","Type":"ContainerDied","Data":"ff2db3badaaad33ce1b102cc6473b51bdcca42488ffd5425215d47e18c615e52"} Oct 30 22:41:32.913812 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:32.913778 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:41:33.030988 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.030951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031057 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031177 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031143 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031195 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031223 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031273 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031304 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031361 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\") pod \"09df02e4-d0fd-4186-b605-02932d5739d1\" (UID: \"09df02e4-d0fd-4186-b605-02932d5739d1\") " Oct 30 22:41:33.031423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031374 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.031740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031468 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.031740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031487 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.031740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031713 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.031945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:33.032011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031950 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.032011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031974 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.032011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.031987 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.032011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.032001 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.033328 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.033299 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.033531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.033500 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:33.033531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.033520 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:33.033673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.033533 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls" pod "09df02e4-d0fd-4186-b605-02932d5739d1" (UID: "09df02e4-d0fd-4186-b605-02932d5739d1"). InnerVolumeSpecName "kube-api-access-zdrls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:33.133175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.133141 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df02e4-d0fd-4186-b605-02932d5739d1-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.133175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.133173 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df02e4-d0fd-4186-b605-02932d5739d1-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.133175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.133182 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.133427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.133192 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df02e4-d0fd-4186-b605-02932d5739d1-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.133427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.133202 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrls\" (UniqueName: \"kubernetes.io/projected/09df02e4-d0fd-4186-b605-02932d5739d1-kube-api-access-zdrls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.518525 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.518482 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" path="/var/lib/kubelet/pods/0915ef56-8ab1-4e37-a043-13f267ad87c4/volumes" Oct 30 22:41:33.519135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.519111 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" path="/var/lib/kubelet/pods/c9896e29-0f85-49f1-94d9-7597e266f71c/volumes" Oct 30 22:41:33.519619 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.519600 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" path="/var/lib/kubelet/pods/fd0bd1c2-3271-43d8-a9e8-d506784e77e1/volumes" Oct 30 22:41:33.797039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.796954 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" Oct 30 22:41:33.797039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.796984 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc" event={"ID":"09df02e4-d0fd-4186-b605-02932d5739d1","Type":"ContainerDied","Data":"349ca584dbc3710f1977a713e7890552dd29ea65ce23ea63cd241519d616e7a3"} Oct 30 22:41:33.797579 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.797053 4305 scope.go:122] "RemoveContainer" containerID="ff2db3badaaad33ce1b102cc6473b51bdcca42488ffd5425215d47e18c615e52" Oct 30 22:41:33.998745 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:33.998710 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:41:34.045060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:34.045026 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-lwbgc"] Oct 30 22:41:35.518775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:35.518735 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" path="/var/lib/kubelet/pods/09df02e4-d0fd-4186-b605-02932d5739d1/volumes" Oct 30 22:41:37.829065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.829033 4305 generic.go:356] "Generic (PLEG): container finished" podID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerID="5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550" exitCode=0 Oct 30 22:41:37.829503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.829109 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerDied","Data":"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550"} Oct 30 22:41:37.829564 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.829539 4305 scope.go:122] "RemoveContainer" containerID="5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550" Oct 30 22:41:37.831082 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.831056 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:37.831147 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.831125 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:37.831183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:37.831160 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:41:38.837050 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:38.837010 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerStarted","Data":"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1"} Oct 30 22:41:39.019707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:39.019672 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:41:39.021318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:39.021275 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:39.021637 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:39.021604 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:40.018596 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:40.018559 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:40.019013 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:40.018613 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:40.835715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:40.835686 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:41:41.021306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:41.019102 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:41.021306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:41.019163 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:42.018578 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:42.018547 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:42.018759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:42.018598 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:43.019065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:43.019022 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:43.019562 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:43.019102 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:44.019220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:44.019182 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:41:44.019812 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:44.019758 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:41:45.022849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:45.022793 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:41:45.023901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:41:45.023878 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:41:46.109320 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:41:46.109280 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:42:00.358885 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.358845 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:42:00.359505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359487 4305 cpu_manager.go:414] "containerMap: removing container" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" Oct 30 22:42:00.359505 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359506 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359520 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359525 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359534 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359539 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359566 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" containerName="maas-api" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359571 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" containerName="maas-api" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359578 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" Oct 30 22:42:00.359595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359583 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" Oct 30 22:42:00.359898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359663 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="09df02e4-d0fd-4186-b605-02932d5739d1" containerName="istio-proxy" Oct 30 22:42:00.359898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359674 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9896e29-0f85-49f1-94d9-7597e266f71c" containerName="istio-proxy" Oct 30 22:42:00.359898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359681 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0915ef56-8ab1-4e37-a043-13f267ad87c4" containerName="istio-proxy" Oct 30 22:42:00.359898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359691 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca71e3e9-ada3-4096-a730-504f508521cf" containerName="maas-api" Oct 30 22:42:00.359898 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.359698 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0bd1c2-3271-43d8-a9e8-d506784e77e1" containerName="istio-proxy" Oct 30 22:42:00.363196 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.363178 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.414574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.414532 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-afb98c\"/\"kube-root-ca.crt\"" Oct 30 22:42:00.460865 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.460836 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-afb98c\"/\"openshift-service-ca.crt\"" Oct 30 22:42:00.461026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.461009 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-afb98c\"/\"default-dockercfg-fnsgl\"" Oct 30 22:42:00.461279 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.461258 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-afb98c\"/\"test-model-afb98c-kserve-self-signed-certs\"" Oct 30 22:42:00.463926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.463894 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:42:00.482961 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.482938 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6kst\" (UniqueName: \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.483089 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.483067 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.483163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.483123 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.483163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.483154 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.483284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.483266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.483324 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.483295 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.584768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584733 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6kst\" (UniqueName: \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.584992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584867 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.584992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584908 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.584992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584929 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.584992 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584970 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.585200 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.584999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.585401 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.585368 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.585512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.585400 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.585512 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.585486 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.587234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.587213 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.587580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.587561 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.647012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.646955 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6kst\" (UniqueName: \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\") pod \"test-model-afb98c-kserve-7bbccc7d9c-62fbj\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:00.675936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:00.675907 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:01.079215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:01.079179 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:42:01.085538 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:42:01.085499 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a550924_4a8a_4e4f_91a5_c3916688b6a1.slice/crio-4f3a184b94b768b103f2bdfb3398fb22d2cd7e58121b09550e01147e229126da WatchSource:0}: Error finding container 4f3a184b94b768b103f2bdfb3398fb22d2cd7e58121b09550e01147e229126da: Status 404 returned error can't find the container with id 4f3a184b94b768b103f2bdfb3398fb22d2cd7e58121b09550e01147e229126da Oct 30 22:42:01.994405 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:01.994363 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" event={"ID":"9a550924-4a8a-4e4f-91a5-c3916688b6a1","Type":"ContainerStarted","Data":"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0"} Oct 30 22:42:01.994792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:01.994419 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" event={"ID":"9a550924-4a8a-4e4f-91a5-c3916688b6a1","Type":"ContainerStarted","Data":"4f3a184b94b768b103f2bdfb3398fb22d2cd7e58121b09550e01147e229126da"} Oct 30 22:42:02.106476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:02.106427 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" podStartSLOduration=2.10641443 podStartE2EDuration="2.10641443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:02.105492726 +0000 UTC m=+2442.700581032" watchObservedRunningTime="2026-10-30 22:42:02.10641443 +0000 UTC m=+2442.701502771" Oct 30 22:42:02.676368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:02.676340 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:03.336577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:03.336541 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:42:10.682119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:10.682085 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:10.699131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:10.699104 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:11.062731 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:11.062700 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:42:16.109559 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:42:16.109520 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:42:18.413427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:18.413377 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" containerID="cri-o://875ecaf05942dac8e922b58702d274c18d25aed163d37c49b4e9f0688314c4bf" gracePeriod=45 Oct 30 22:42:19.306888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.306223 4305 generic.go:356] "Generic (PLEG): container finished" podID="998636d1-6870-4142-becc-d4b6c74795ce" containerID="875ecaf05942dac8e922b58702d274c18d25aed163d37c49b4e9f0688314c4bf" exitCode=0 Oct 30 22:42:19.306888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.306311 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" event={"ID":"998636d1-6870-4142-becc-d4b6c74795ce","Type":"ContainerDied","Data":"875ecaf05942dac8e922b58702d274c18d25aed163d37c49b4e9f0688314c4bf"} Oct 30 22:42:19.623837 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.623787 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648242 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648303 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648402 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648433 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648520 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.678822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.648566 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\" (UniqueName: \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\") pod \"998636d1-6870-4142-becc-d4b6c74795ce\" (UID: \"998636d1-6870-4142-becc-d4b6c74795ce\") " Oct 30 22:42:19.715154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.695927 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.715154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.708271 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.718473 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.718429 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.725823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.722930 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.750910 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.754765 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.754902 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.754922 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.754936 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998636d1-6870-4142-becc-d4b6c74795ce-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.754950 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998636d1-6870-4142-becc-d4b6c74795ce-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.783743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.758119 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2" pod "998636d1-6870-4142-becc-d4b6c74795ce" (UID: "998636d1-6870-4142-becc-d4b6c74795ce"). InnerVolumeSpecName "kube-api-access-sszj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:19.856142 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:19.856026 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sszj2\" (UniqueName: \"kubernetes.io/projected/998636d1-6870-4142-becc-d4b6c74795ce-kube-api-access-sszj2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:20.199998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.199949 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:42:20.315417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.315385 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" Oct 30 22:42:20.315417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.315407 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694" event={"ID":"998636d1-6870-4142-becc-d4b6c74795ce","Type":"ContainerDied","Data":"c49dbb770c5b7ed2441f92076fd38dbd1229dc02d4f7efe737cbade3532595fa"} Oct 30 22:42:20.315633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.315458 4305 scope.go:122] "RemoveContainer" containerID="875ecaf05942dac8e922b58702d274c18d25aed163d37c49b4e9f0688314c4bf" Oct 30 22:42:20.473347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.473239 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:42:20.516423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:20.516397 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/simulator-w6-be17e7-kserve-66dc55c8f4-6h694"] Oct 30 22:42:21.518510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:42:21.518478 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998636d1-6870-4142-becc-d4b6c74795ce" path="/var/lib/kubelet/pods/998636d1-6870-4142-becc-d4b6c74795ce/volumes" Oct 30 22:42:46.110557 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:42:46.110464 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:43:07.100306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.100272 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:43:07.100700 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.100501 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" podUID="8ad7b205-65c7-4f23-8033-64dc828dda1b" containerName="maas-api" containerID="cri-o://6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91" gracePeriod=30 Oct 30 22:43:07.413457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.413430 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:43:07.487104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.486007 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\" (UniqueName: \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\") pod \"8ad7b205-65c7-4f23-8033-64dc828dda1b\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " Oct 30 22:43:07.487104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.486077 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\") pod \"8ad7b205-65c7-4f23-8033-64dc828dda1b\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " Oct 30 22:43:07.487104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.486158 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") pod \"8ad7b205-65c7-4f23-8033-64dc828dda1b\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " Oct 30 22:43:07.487104 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.486218 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\") pod \"8ad7b205-65c7-4f23-8033-64dc828dda1b\" (UID: \"8ad7b205-65c7-4f23-8033-64dc828dda1b\") " Oct 30 22:43:07.487959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.487597 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca" pod "8ad7b205-65c7-4f23-8033-64dc828dda1b" (UID: "8ad7b205-65c7-4f23-8033-64dc828dda1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:07.497318 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.497275 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls" pod "8ad7b205-65c7-4f23-8033-64dc828dda1b" (UID: "8ad7b205-65c7-4f23-8033-64dc828dda1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:07.504089 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.503781 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs" pod "8ad7b205-65c7-4f23-8033-64dc828dda1b" (UID: "8ad7b205-65c7-4f23-8033-64dc828dda1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:07.504894 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.504865 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt" pod "8ad7b205-65c7-4f23-8033-64dc828dda1b" (UID: "8ad7b205-65c7-4f23-8033-64dc828dda1b"). InnerVolumeSpecName "kube-api-access-k79jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:07.582661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.582628 4305 generic.go:356] "Generic (PLEG): container finished" podID="8ad7b205-65c7-4f23-8033-64dc828dda1b" containerID="6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91" exitCode=0 Oct 30 22:43:07.582875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.582714 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" Oct 30 22:43:07.582875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.582718 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" event={"ID":"8ad7b205-65c7-4f23-8033-64dc828dda1b","Type":"ContainerDied","Data":"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91"} Oct 30 22:43:07.582875 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.582838 4305 scope.go:122] "RemoveContainer" containerID="6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91" Oct 30 22:43:07.583026 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.582830 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86" event={"ID":"8ad7b205-65c7-4f23-8033-64dc828dda1b","Type":"ContainerDied","Data":"8105679fd7930b3350e3cb464641f7879f49d000ab763b0380542a285fc0da63"} Oct 30 22:43:07.588815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.588779 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ad7b205-65c7-4f23-8033-64dc828dda1b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.588951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.588819 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k79jt\" (UniqueName: \"kubernetes.io/projected/8ad7b205-65c7-4f23-8033-64dc828dda1b-kube-api-access-k79jt\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.588951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.588835 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.588951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.588849 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ad7b205-65c7-4f23-8033-64dc828dda1b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.592872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.592853 4305 scope.go:122] "RemoveContainer" containerID="6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91" Oct 30 22:43:07.593174 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:43:07.593155 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91\": container with ID starting with 6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91 not found: ID does not exist" containerID="6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91" Oct 30 22:43:07.593238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.593182 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91"} err="failed to get container status \"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91\": rpc error: code = NotFound desc = could not find container \"6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91\": container with ID starting with 6f8faea80bb001b19b3d963dc19db78899d00477684711e37858deaa1f2a0e91 not found: ID does not exist" Oct 30 22:43:07.737481 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.737448 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:43:07.741880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:07.741854 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-be17e7-867c8c9f58-nrq86"] Oct 30 22:43:09.520080 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:09.520041 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad7b205-65c7-4f23-8033-64dc828dda1b" path="/var/lib/kubelet/pods/8ad7b205-65c7-4f23-8033-64dc828dda1b/volumes" Oct 30 22:43:16.111987 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:43:16.111949 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:43:16.498553 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:16.498518 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:43:16.498784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:16.498761 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" containerID="cri-o://3cd196a0aebdc6817b60c05bafd12eb6f1f2eca3e6ef54eccf0671a49e2d984d" gracePeriod=30 Oct 30 22:43:21.672309 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.672275 4305 generic.go:356] "Generic (PLEG): container finished" podID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerID="3cd196a0aebdc6817b60c05bafd12eb6f1f2eca3e6ef54eccf0671a49e2d984d" exitCode=0 Oct 30 22:43:21.672758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.672340 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerDied","Data":"3cd196a0aebdc6817b60c05bafd12eb6f1f2eca3e6ef54eccf0671a49e2d984d"} Oct 30 22:43:21.672758 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.672385 4305 scope.go:122] "RemoveContainer" containerID="cb03f68f7c008653f6ea76c5d51170582ec3312c69a54a0a1f80650de572c1d4" Oct 30 22:43:21.768598 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.768532 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:43:21.878593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878560 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.878762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878598 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.878762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878678 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.878762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878728 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.878762 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878747 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.879031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878780 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.879031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.879031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.878917 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.879192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879002 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.879192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879016 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\") pod \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\" (UID: \"a48eddfb-85f8-4fc4-aa5a-44dff6778940\") " Oct 30 22:43:21.879192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879095 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.879192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879008 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:21.879408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879209 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.879408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879329 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.879726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879632 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.879726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879686 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.879726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879704 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.879726 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879718 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.880038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.879735 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.881130 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.881097 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.881227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.881156 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:21.881227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.881179 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "kube-api-access-5m8mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:21.881227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.881192 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token" pod "a48eddfb-85f8-4fc4-aa5a-44dff6778940" (UID: "a48eddfb-85f8-4fc4-aa5a-44dff6778940"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:21.980843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.980794 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.980843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.980839 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.981047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.980850 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8mp\" (UniqueName: \"kubernetes.io/projected/a48eddfb-85f8-4fc4-aa5a-44dff6778940-kube-api-access-5m8mp\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.981047 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:21.980863 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a48eddfb-85f8-4fc4-aa5a-44dff6778940-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:22.680690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:22.680656 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" event={"ID":"a48eddfb-85f8-4fc4-aa5a-44dff6778940","Type":"ContainerDied","Data":"36d542fc58fdb7b830e16d9a40da6bc693751b7df20990bb165d221a4a198df4"} Oct 30 22:43:22.681173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:22.680717 4305 scope.go:122] "RemoveContainer" containerID="3cd196a0aebdc6817b60c05bafd12eb6f1f2eca3e6ef54eccf0671a49e2d984d" Oct 30 22:43:22.681173 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:22.680669 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26" Oct 30 22:43:22.889749 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:22.889714 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:43:22.935647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:22.935571 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-tkx26"] Oct 30 22:43:23.518492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:23.518458 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" path="/var/lib/kubelet/pods/a48eddfb-85f8-4fc4-aa5a-44dff6778940/volumes" Oct 30 22:43:46.109713 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:43:46.109678 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:43:49.086703 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.086673 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:43:49.087152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.086873 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerName="maas-api" containerID="cri-o://dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec" gracePeriod=30 Oct 30 22:43:49.195187 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.195144 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.116:8443/health\": dial tcp 10.134.0.116:8443: connect: connection refused" Oct 30 22:43:49.387129 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.387105 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:43:49.440766 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.440735 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") pod \"2d6f6d42-621c-4f0e-b194-118512ee0366\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " Oct 30 22:43:49.440895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.440789 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\") pod \"2d6f6d42-621c-4f0e-b194-118512ee0366\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " Oct 30 22:43:49.440895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.440852 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\") pod \"2d6f6d42-621c-4f0e-b194-118512ee0366\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " Oct 30 22:43:49.440895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.440885 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\" (UniqueName: \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\") pod \"2d6f6d42-621c-4f0e-b194-118512ee0366\" (UID: \"2d6f6d42-621c-4f0e-b194-118512ee0366\") " Oct 30 22:43:49.441210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.441179 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca" pod "2d6f6d42-621c-4f0e-b194-118512ee0366" (UID: "2d6f6d42-621c-4f0e-b194-118512ee0366"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:49.441511 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.441469 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6f6d42-621c-4f0e-b194-118512ee0366-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.443003 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.442963 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n" pod "2d6f6d42-621c-4f0e-b194-118512ee0366" (UID: "2d6f6d42-621c-4f0e-b194-118512ee0366"). InnerVolumeSpecName "kube-api-access-56w4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:49.443116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.443022 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs" pod "2d6f6d42-621c-4f0e-b194-118512ee0366" (UID: "2d6f6d42-621c-4f0e-b194-118512ee0366"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:49.443116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.443057 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls" pod "2d6f6d42-621c-4f0e-b194-118512ee0366" (UID: "2d6f6d42-621c-4f0e-b194-118512ee0366"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:49.542608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.542576 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.542608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.542608 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d6f6d42-621c-4f0e-b194-118512ee0366-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.542847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.542626 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56w4n\" (UniqueName: \"kubernetes.io/projected/2d6f6d42-621c-4f0e-b194-118512ee0366-kube-api-access-56w4n\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:49.834698 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.834663 4305 generic.go:356] "Generic (PLEG): container finished" podID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerID="dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec" exitCode=0 Oct 30 22:43:49.834903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.834728 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" event={"ID":"2d6f6d42-621c-4f0e-b194-118512ee0366","Type":"ContainerDied","Data":"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec"} Oct 30 22:43:49.834903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.834745 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" Oct 30 22:43:49.834903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.834768 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6" event={"ID":"2d6f6d42-621c-4f0e-b194-118512ee0366","Type":"ContainerDied","Data":"240dde4f7545d3905abf43722b499b550676521f5142d6f95e42aa2fef65c301"} Oct 30 22:43:49.834903 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.834789 4305 scope.go:122] "RemoveContainer" containerID="dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec" Oct 30 22:43:49.845523 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.845498 4305 scope.go:122] "RemoveContainer" containerID="dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec" Oct 30 22:43:49.845867 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:43:49.845845 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec\": container with ID starting with dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec not found: ID does not exist" containerID="dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec" Oct 30 22:43:49.845971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:49.845873 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec"} err="failed to get container status \"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec\": rpc error: code = NotFound desc = could not find container \"dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec\": container with ID starting with dff90d7265d623c90a42a38170a0e5940a43e54638e317974a74cb7dae7c8aec not found: ID does not exist" Oct 30 22:43:50.014094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:50.014050 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:43:50.038550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:50.038500 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771ff9-5f4d546885-945t6"] Oct 30 22:43:51.518397 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:51.518366 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" path="/var/lib/kubelet/pods/2d6f6d42-621c-4f0e-b194-118512ee0366/volumes" Oct 30 22:43:58.806055 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:58.806018 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:43:58.806461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:43:58.806266 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" containerID="cri-o://a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c" gracePeriod=30 Oct 30 22:44:04.075018 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.074950 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:44:04.130243 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130214 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130320 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130354 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130393 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130383 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130555 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130538 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130610 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130569 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130664 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130635 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130721 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130679 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130774 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130725 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\") pod \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\" (UID: \"3f3d1b90-8506-493c-bc2a-3e79d21bba50\") " Oct 30 22:44:04.130880 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130854 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:44:04.130959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130841 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:04.130959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130921 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:04.130959 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.130940 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:04.131131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131103 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:04.131347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131320 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.131491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131350 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.131491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131367 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.131491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131380 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.131491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.131395 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.132595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.132568 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:44:04.132680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.132613 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "kube-api-access-glz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:04.132680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.132664 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:44:04.132891 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.132871 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy" pod "3f3d1b90-8506-493c-bc2a-3e79d21bba50" (UID: "3f3d1b90-8506-493c-bc2a-3e79d21bba50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:44:04.233000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.232972 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz85\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-kube-api-access-glz85\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.233117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.233012 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.233117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.233022 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.233117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.233032 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3d1b90-8506-493c-bc2a-3e79d21bba50-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:44:04.923639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.923604 4305 generic.go:356] "Generic (PLEG): container finished" podID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerID="a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c" exitCode=0 Oct 30 22:44:04.923863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.923656 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" Oct 30 22:44:04.923863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.923661 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" event={"ID":"3f3d1b90-8506-493c-bc2a-3e79d21bba50","Type":"ContainerDied","Data":"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c"} Oct 30 22:44:04.923863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.923776 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f" event={"ID":"3f3d1b90-8506-493c-bc2a-3e79d21bba50","Type":"ContainerDied","Data":"6858e09498726a02d46dce55fce2ac3c9345f67e58b3cf5331f96bc4f9f26e41"} Oct 30 22:44:04.923863 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.923822 4305 scope.go:122] "RemoveContainer" containerID="a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c" Oct 30 22:44:04.933413 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.933391 4305 scope.go:122] "RemoveContainer" containerID="a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c" Oct 30 22:44:04.933665 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:44:04.933649 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c\": container with ID starting with a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c not found: ID does not exist" containerID="a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c" Oct 30 22:44:04.933723 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:04.933673 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c"} err="failed to get container status \"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c\": rpc error: code = NotFound desc = could not find container \"a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c\": container with ID starting with a0497f1e922b35579a350d3b940060bb3c56ade5685fd67e4bd3b5b44126e96c not found: ID does not exist" Oct 30 22:44:05.124543 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:05.124514 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:44:05.170492 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:05.170464 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-ldv4f"] Oct 30 22:44:05.518602 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:44:05.518558 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" path="/var/lib/kubelet/pods/3f3d1b90-8506-493c-bc2a-3e79d21bba50/volumes" Oct 30 22:44:16.110273 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:44:16.110236 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:44:46.110664 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:44:46.110627 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:45:16.112232 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:16.112195 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:45:29.440507 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:29.440468 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:45:29.441021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:29.440750 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" containerID="cri-o://c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79" gracePeriod=30 Oct 30 22:45:34.694584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.694559 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:45:34.767261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767196 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767238 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767283 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767427 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767401 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767520 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767470 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767545 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767534 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:34.767679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767579 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:34.767679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767620 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767679 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767648 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767672 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\") pod \"324da313-68f4-4270-9984-91d39e6c15c6\" (UID: \"324da313-68f4-4270-9984-91d39e6c15c6\") " Oct 30 22:45:34.767904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767762 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:34.767904 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.767845 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:34.768255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768068 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:34.768372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768260 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324da313-68f4-4270-9984-91d39e6c15c6-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.768372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768286 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.768372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768301 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.768372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768315 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.768372 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.768329 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.770065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.770041 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:34.770153 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.770134 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:45:34.770213 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.770163 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:34.770253 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.770230 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft" pod "324da313-68f4-4270-9984-91d39e6c15c6" (UID: "324da313-68f4-4270-9984-91d39e6c15c6"). InnerVolumeSpecName "kube-api-access-l97ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:34.869533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.869497 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324da313-68f4-4270-9984-91d39e6c15c6-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.869820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.869783 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.869970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.869955 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324da313-68f4-4270-9984-91d39e6c15c6-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:34.870073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:34.870059 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97ft\" (UniqueName: \"kubernetes.io/projected/324da313-68f4-4270-9984-91d39e6c15c6-kube-api-access-l97ft\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:35.431234 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.431191 4305 generic.go:356] "Generic (PLEG): container finished" podID="324da313-68f4-4270-9984-91d39e6c15c6" containerID="c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79" exitCode=0 Oct 30 22:45:35.431429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.431244 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerDied","Data":"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79"} Oct 30 22:45:35.431429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.431257 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" Oct 30 22:45:35.431429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.431315 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78" event={"ID":"324da313-68f4-4270-9984-91d39e6c15c6","Type":"ContainerDied","Data":"5a70a04f123440e60897beab123909ae8dc587175858fd179c8559d0fc889fff"} Oct 30 22:45:35.431429 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.431337 4305 scope.go:122] "RemoveContainer" containerID="c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79" Oct 30 22:45:35.441296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.441281 4305 scope.go:122] "RemoveContainer" containerID="adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef" Oct 30 22:45:35.449712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.449694 4305 scope.go:122] "RemoveContainer" containerID="c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79" Oct 30 22:45:35.449977 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:35.449952 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79\": container with ID starting with c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79 not found: ID does not exist" containerID="c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79" Oct 30 22:45:35.450031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.449993 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79"} err="failed to get container status \"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79\": rpc error: code = NotFound desc = could not find container \"c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79\": container with ID starting with c72ed3900c3a82be7e678ee8e7e3ca8b8ab34800349f839d8a413004eddc8a79 not found: ID does not exist" Oct 30 22:45:35.450031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.450012 4305 scope.go:122] "RemoveContainer" containerID="adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef" Oct 30 22:45:35.450240 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:35.450223 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef\": container with ID starting with adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef not found: ID does not exist" containerID="adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef" Oct 30 22:45:35.450284 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.450245 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef"} err="failed to get container status \"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef\": rpc error: code = NotFound desc = could not find container \"adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef\": container with ID starting with adb73da2596a7a8681e16078be47adb06d76cf1795dc807c1fc1bfe4fcbee4ef not found: ID does not exist" Oct 30 22:45:35.596075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.596035 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:45:35.602239 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:35.602206 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-96t78"] Oct 30 22:45:37.518316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:37.518284 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324da313-68f4-4270-9984-91d39e6c15c6" path="/var/lib/kubelet/pods/324da313-68f4-4270-9984-91d39e6c15c6/volumes" Oct 30 22:45:44.013606 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:44.013573 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:45:44.014263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:44.013816 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" containerID="cri-o://d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1" gracePeriod=30 Oct 30 22:45:45.025878 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:45.025842 4305 patch_prober.go:29] interesting pod/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:45:45.026268 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:45.025903 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:45:46.109721 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:46.109680 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:45:49.273261 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.273234 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:45:49.362934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.362850 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.362934 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.362918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.362958 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363007 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363036 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363161 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363127 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363163 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363247 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\") pod \"14cede09-a059-4eda-9c3c-f7f45097a47e\" (UID: \"14cede09-a059-4eda-9c3c-f7f45097a47e\") " Oct 30 22:45:49.363432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363334 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.363432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363346 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.363432 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363371 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.363689 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363579 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.363738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363682 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:49.363854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363789 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.363935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363859 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.363935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363875 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.363935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.363888 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.365303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.365274 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:45:49.365408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.365303 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:49.365565 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.365551 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "kube-api-access-c57jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:49.365690 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.365667 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy" pod "14cede09-a059-4eda-9c3c-f7f45097a47e" (UID: "14cede09-a059-4eda-9c3c-f7f45097a47e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.465152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.465115 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.465398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.465158 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57jb\" (UniqueName: \"kubernetes.io/projected/14cede09-a059-4eda-9c3c-f7f45097a47e-kube-api-access-c57jb\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.465398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.465175 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.465398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.465190 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14cede09-a059-4eda-9c3c-f7f45097a47e-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.465398 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.465207 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14cede09-a059-4eda-9c3c-f7f45097a47e-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.511238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.511201 4305 generic.go:356] "Generic (PLEG): container finished" podID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerID="d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1" exitCode=0 Oct 30 22:45:49.513855 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.513829 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" Oct 30 22:45:49.519077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.519035 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerDied","Data":"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1"} Oct 30 22:45:49.519183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.519107 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q" event={"ID":"14cede09-a059-4eda-9c3c-f7f45097a47e","Type":"ContainerDied","Data":"2def4feba663f74a3da58d1ff411628c928978895e4f0e7eeab20b0c9506561e"} Oct 30 22:45:49.519183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.519142 4305 scope.go:122] "RemoveContainer" containerID="d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1" Oct 30 22:45:49.529022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.528996 4305 scope.go:122] "RemoveContainer" containerID="5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550" Oct 30 22:45:49.537899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.537870 4305 scope.go:122] "RemoveContainer" containerID="d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1" Oct 30 22:45:49.538197 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:49.538172 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1\": container with ID starting with d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1 not found: ID does not exist" containerID="d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1" Oct 30 22:45:49.538288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.538205 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1"} err="failed to get container status \"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1\": rpc error: code = NotFound desc = could not find container \"d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1\": container with ID starting with d8416ee215ac26c78e9514d70e3b1cbb7428c9b6b841ff9aa47b135f99b3f5f1 not found: ID does not exist" Oct 30 22:45:49.538288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.538223 4305 scope.go:122] "RemoveContainer" containerID="5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550" Oct 30 22:45:49.538575 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:45:49.538553 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550\": container with ID starting with 5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550 not found: ID does not exist" containerID="5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550" Oct 30 22:45:49.538654 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.538584 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550"} err="failed to get container status \"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550\": rpc error: code = NotFound desc = could not find container \"5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550\": container with ID starting with 5ae9fe13e6476b62f35f444a2ecede07720437876d2e73cb7a8f2fb2afc2f550 not found: ID does not exist" Oct 30 22:45:49.667951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.667868 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:45:49.672478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:49.672450 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9fce61-openshift-default-56798fcc9-c8n4q"] Oct 30 22:45:51.520244 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:45:51.520205 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" path="/var/lib/kubelet/pods/14cede09-a059-4eda-9c3c-f7f45097a47e/volumes" Oct 30 22:46:13.632826 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:13.632766 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:46:13.649135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:13.649091 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" podUID="cc77f4d4-5742-45c6-a838-c316551364f5" containerName="maas-api" containerID="cri-o://207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e" gracePeriod=30 Oct 30 22:46:13.934997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:13.934973 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:46:14.079844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.079809 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\" (UniqueName: \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\") pod \"cc77f4d4-5742-45c6-a838-c316551364f5\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " Oct 30 22:46:14.080070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.079898 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\") pod \"cc77f4d4-5742-45c6-a838-c316551364f5\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " Oct 30 22:46:14.080070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.079956 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\") pod \"cc77f4d4-5742-45c6-a838-c316551364f5\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " Oct 30 22:46:14.080070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.080027 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") pod \"cc77f4d4-5742-45c6-a838-c316551364f5\" (UID: \"cc77f4d4-5742-45c6-a838-c316551364f5\") " Oct 30 22:46:14.081249 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.081212 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca" pod "cc77f4d4-5742-45c6-a838-c316551364f5" (UID: "cc77f4d4-5742-45c6-a838-c316551364f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:14.082715 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.082690 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2" pod "cc77f4d4-5742-45c6-a838-c316551364f5" (UID: "cc77f4d4-5742-45c6-a838-c316551364f5"). InnerVolumeSpecName "kube-api-access-rnsg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:14.083021 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.082999 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs" pod "cc77f4d4-5742-45c6-a838-c316551364f5" (UID: "cc77f4d4-5742-45c6-a838-c316551364f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:14.083105 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.083040 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls" pod "cc77f4d4-5742-45c6-a838-c316551364f5" (UID: "cc77f4d4-5742-45c6-a838-c316551364f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:14.181709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.181633 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:14.181709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.181665 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsg2\" (UniqueName: \"kubernetes.io/projected/cc77f4d4-5742-45c6-a838-c316551364f5-kube-api-access-rnsg2\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:14.181709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.181675 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc77f4d4-5742-45c6-a838-c316551364f5-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:14.181709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.181684 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc77f4d4-5742-45c6-a838-c316551364f5-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:14.656511 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.656476 4305 generic.go:356] "Generic (PLEG): container finished" podID="cc77f4d4-5742-45c6-a838-c316551364f5" containerID="207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e" exitCode=0 Oct 30 22:46:14.656948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.656530 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" Oct 30 22:46:14.656948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.656562 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" event={"ID":"cc77f4d4-5742-45c6-a838-c316551364f5","Type":"ContainerDied","Data":"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e"} Oct 30 22:46:14.656948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.656624 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885" event={"ID":"cc77f4d4-5742-45c6-a838-c316551364f5","Type":"ContainerDied","Data":"a59b341287892d7d0ff6c3e4c139d65bb25db5f0a19173a792663f45b85ddfaa"} Oct 30 22:46:14.656948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.656647 4305 scope.go:122] "RemoveContainer" containerID="207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e" Oct 30 22:46:14.667488 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.667471 4305 scope.go:122] "RemoveContainer" containerID="207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e" Oct 30 22:46:14.667742 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:46:14.667724 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e\": container with ID starting with 207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e not found: ID does not exist" containerID="207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e" Oct 30 22:46:14.667809 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.667749 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e"} err="failed to get container status \"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e\": rpc error: code = NotFound desc = could not find container \"207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e\": container with ID starting with 207d24633d201f00b716395ec38a3c4e7c857dfb51df1209271a4563e196c16e not found: ID does not exist" Oct 30 22:46:14.811839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.811806 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:46:14.855476 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:14.855438 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6b55f-bf789b944-6z885"] Oct 30 22:46:15.518674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:15.518637 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc77f4d4-5742-45c6-a838-c316551364f5" path="/var/lib/kubelet/pods/cc77f4d4-5742-45c6-a838-c316551364f5/volumes" Oct 30 22:46:16.110445 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:46:16.110407 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:46:19.889584 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:19.889555 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:46:19.893098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:19.893076 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:46:46.109206 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:46:46.109169 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:46:47.321738 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.321708 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:46:47.322151 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.321915 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" podUID="0e95a88c-18da-4a65-b471-20528f5a7cc6" containerName="maas-api" containerID="cri-o://0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa" gracePeriod=30 Oct 30 22:46:47.616092 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.616068 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:46:47.743184 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.743148 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") pod \"0e95a88c-18da-4a65-b471-20528f5a7cc6\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " Oct 30 22:46:47.743388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.743260 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\") pod \"0e95a88c-18da-4a65-b471-20528f5a7cc6\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " Oct 30 22:46:47.743388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.743299 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\") pod \"0e95a88c-18da-4a65-b471-20528f5a7cc6\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " Oct 30 22:46:47.743388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.743333 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\" (UniqueName: \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\") pod \"0e95a88c-18da-4a65-b471-20528f5a7cc6\" (UID: \"0e95a88c-18da-4a65-b471-20528f5a7cc6\") " Oct 30 22:46:47.743927 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.743882 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca" pod "0e95a88c-18da-4a65-b471-20528f5a7cc6" (UID: "0e95a88c-18da-4a65-b471-20528f5a7cc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:47.745346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.745314 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls" pod "0e95a88c-18da-4a65-b471-20528f5a7cc6" (UID: "0e95a88c-18da-4a65-b471-20528f5a7cc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:47.745480 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.745402 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77" pod "0e95a88c-18da-4a65-b471-20528f5a7cc6" (UID: "0e95a88c-18da-4a65-b471-20528f5a7cc6"). InnerVolumeSpecName "kube-api-access-8jf77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:47.745789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.745766 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs" pod "0e95a88c-18da-4a65-b471-20528f5a7cc6" (UID: "0e95a88c-18da-4a65-b471-20528f5a7cc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:47.856430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.856284 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:47.856430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.856321 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e95a88c-18da-4a65-b471-20528f5a7cc6-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:47.856430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.856337 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e95a88c-18da-4a65-b471-20528f5a7cc6-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:47.856430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.856353 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jf77\" (UniqueName: \"kubernetes.io/projected/0e95a88c-18da-4a65-b471-20528f5a7cc6-kube-api-access-8jf77\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:47.913106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.913067 4305 generic.go:356] "Generic (PLEG): container finished" podID="0e95a88c-18da-4a65-b471-20528f5a7cc6" containerID="0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa" exitCode=0 Oct 30 22:46:47.913255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.913114 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" Oct 30 22:46:47.913255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.913144 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" event={"ID":"0e95a88c-18da-4a65-b471-20528f5a7cc6","Type":"ContainerDied","Data":"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa"} Oct 30 22:46:47.913255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.913190 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh" event={"ID":"0e95a88c-18da-4a65-b471-20528f5a7cc6","Type":"ContainerDied","Data":"cc3a9af71f47a1cc47634234141a2ffbc89954ee9036ee63310f113d1a0c83bc"} Oct 30 22:46:47.913255 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.913211 4305 scope.go:122] "RemoveContainer" containerID="0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa" Oct 30 22:46:47.922644 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.922625 4305 scope.go:122] "RemoveContainer" containerID="0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa" Oct 30 22:46:47.922935 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:46:47.922915 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa\": container with ID starting with 0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa not found: ID does not exist" containerID="0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa" Oct 30 22:46:47.923006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:47.922944 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa"} err="failed to get container status \"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa\": rpc error: code = NotFound desc = could not find container \"0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa\": container with ID starting with 0a09ee2cc63328673e04fdd87ec27310c280ea96f93bd81dd64c828a11c419aa not found: ID does not exist" Oct 30 22:46:48.069993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:48.069964 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:46:48.074757 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:48.074727 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23f3d4-6c59d8fd66-2zwhh"] Oct 30 22:46:49.518751 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:46:49.518716 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e95a88c-18da-4a65-b471-20528f5a7cc6" path="/var/lib/kubelet/pods/0e95a88c-18da-4a65-b471-20528f5a7cc6/volumes" Oct 30 22:47:02.375349 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.375261 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:47:02.375929 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.375500 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" podUID="be96707f-c6f0-4313-9543-fd586492afa5" containerName="maas-api" containerID="cri-o://82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6" gracePeriod=30 Oct 30 22:47:02.447723 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.447686 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:47:02.491406 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.491365 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" podUID="be96707f-c6f0-4313-9543-fd586492afa5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.117:8443/health\": dial tcp 10.134.0.117:8443: connect: connection refused" Oct 30 22:47:02.684450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.684423 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:47:02.740935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.739842 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\") pod \"be96707f-c6f0-4313-9543-fd586492afa5\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " Oct 30 22:47:02.740935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.739951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") pod \"be96707f-c6f0-4313-9543-fd586492afa5\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " Oct 30 22:47:02.740935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.740067 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\" (UniqueName: \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\") pod \"be96707f-c6f0-4313-9543-fd586492afa5\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " Oct 30 22:47:02.740935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.740123 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\") pod \"be96707f-c6f0-4313-9543-fd586492afa5\" (UID: \"be96707f-c6f0-4313-9543-fd586492afa5\") " Oct 30 22:47:02.742210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.742173 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca" pod "be96707f-c6f0-4313-9543-fd586492afa5" (UID: "be96707f-c6f0-4313-9543-fd586492afa5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:02.743857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.743829 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t" pod "be96707f-c6f0-4313-9543-fd586492afa5" (UID: "be96707f-c6f0-4313-9543-fd586492afa5"). InnerVolumeSpecName "kube-api-access-7p47t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:02.743857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.743843 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs" pod "be96707f-c6f0-4313-9543-fd586492afa5" (UID: "be96707f-c6f0-4313-9543-fd586492afa5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:02.744001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.743836 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls" pod "be96707f-c6f0-4313-9543-fd586492afa5" (UID: "be96707f-c6f0-4313-9543-fd586492afa5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:02.841278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.841242 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p47t\" (UniqueName: \"kubernetes.io/projected/be96707f-c6f0-4313-9543-fd586492afa5-kube-api-access-7p47t\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:02.841278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.841273 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:02.841278 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.841283 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be96707f-c6f0-4313-9543-fd586492afa5-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:02.841618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.841294 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be96707f-c6f0-4313-9543-fd586492afa5-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:02.992940 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.992900 4305 generic.go:356] "Generic (PLEG): container finished" podID="be96707f-c6f0-4313-9543-fd586492afa5" containerID="82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6" exitCode=0 Oct 30 22:47:02.993119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.992952 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" Oct 30 22:47:02.993119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.992984 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" event={"ID":"be96707f-c6f0-4313-9543-fd586492afa5","Type":"ContainerDied","Data":"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6"} Oct 30 22:47:02.993119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.993045 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p" event={"ID":"be96707f-c6f0-4313-9543-fd586492afa5","Type":"ContainerDied","Data":"5a3cde5e63c546eb8b7d646fdd34dcf62e50a8e51308760a68b5c1f40d92cdcf"} Oct 30 22:47:02.993119 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:02.993071 4305 scope.go:122] "RemoveContainer" containerID="82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6" Oct 30 22:47:03.004302 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:03.004268 4305 scope.go:122] "RemoveContainer" containerID="82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6" Oct 30 22:47:03.004592 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:03.004572 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6\": container with ID starting with 82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6 not found: ID does not exist" containerID="82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6" Oct 30 22:47:03.004640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:03.004605 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6"} err="failed to get container status \"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6\": rpc error: code = NotFound desc = could not find container \"82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6\": container with ID starting with 82f4e4a7fb2906349082480ccb06f83bbccc93d8fd9748fac33c9669871d37e6 not found: ID does not exist" Oct 30 22:47:03.148736 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:03.148700 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:47:03.155306 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:03.155274 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d585d2-7d4c776bf7-7hc5p"] Oct 30 22:47:03.520315 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:03.520276 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be96707f-c6f0-4313-9543-fd586492afa5" path="/var/lib/kubelet/pods/be96707f-c6f0-4313-9543-fd586492afa5/volumes" Oct 30 22:47:11.912813 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:11.912758 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:47:11.913313 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:11.913074 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" containerID="cri-o://78dfb2e1111b58d26d99357b862751f16d0cd2af21f2c1f0069e21636e96ba34" gracePeriod=30 Oct 30 22:47:16.109506 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:16.109467 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:47:17.084781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.084750 4305 generic.go:356] "Generic (PLEG): container finished" podID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerID="78dfb2e1111b58d26d99357b862751f16d0cd2af21f2c1f0069e21636e96ba34" exitCode=0 Oct 30 22:47:17.084967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.084822 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerDied","Data":"78dfb2e1111b58d26d99357b862751f16d0cd2af21f2c1f0069e21636e96ba34"} Oct 30 22:47:17.084967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.084873 4305 scope.go:122] "RemoveContainer" containerID="a35cf403f47bb00621799616a80b187dd1b3275a630b23c6775a2a170e061978" Oct 30 22:47:17.177057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.176996 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:47:17.310999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.310969 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311008 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311107 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311180 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311165 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311186 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311211 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311255 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311251 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311282 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311359 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311338 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\") pod \"d9f12a95-569c-43a1-b631-e9991c57e11f\" (UID: \"d9f12a95-569c-43a1-b631-e9991c57e11f\") " Oct 30 22:47:17.311727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311455 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.311727 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311557 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:17.311887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311772 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.311887 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311848 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.311996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311975 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f12a95-569c-43a1-b631-e9991c57e11f-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.311996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.311992 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.312116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.312006 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.312116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.312019 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.312116 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.312030 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.313527 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.313503 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:17.313639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.313540 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.313639 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.313586 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "kube-api-access-x2jkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:17.313716 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.313671 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token" pod "d9f12a95-569c-43a1-b631-e9991c57e11f" (UID: "d9f12a95-569c-43a1-b631-e9991c57e11f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:17.413687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.413650 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.413687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.413683 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jkq\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-kube-api-access-x2jkq\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.413687 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.413694 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.413980 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.413704 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f12a95-569c-43a1-b631-e9991c57e11f-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.421485 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.421458 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:47:17.468655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.468605 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" podUID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" containerName="main" containerID="cri-o://02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0" gracePeriod=45 Oct 30 22:47:17.710056 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.710028 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:47:17.818011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.817894 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818011 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.817979 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818078 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818109 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818166 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.818263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818188 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818229 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\" (UniqueName: \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\") pod \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\" (UID: \"9a550924-4a8a-4e4f-91a5-c3916688b6a1\") " Oct 30 22:47:17.818661 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818637 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.818789 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.818767 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.819043 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.819021 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.821697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.821671 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "kube-api-access-g6kst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:17.821852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.821792 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:17.825705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.825677 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm" pod "9a550924-4a8a-4e4f-91a5-c3916688b6a1" (UID: "9a550924-4a8a-4e4f-91a5-c3916688b6a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:17.919756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.919721 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.919756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.919753 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.919756 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.919763 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.920054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.919774 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a550924-4a8a-4e4f-91a5-c3916688b6a1-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:17.920054 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:17.919785 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6kst\" (UniqueName: \"kubernetes.io/projected/9a550924-4a8a-4e4f-91a5-c3916688b6a1-kube-api-access-g6kst\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:18.092012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.091928 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" Oct 30 22:47:18.092012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.091946 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw" event={"ID":"d9f12a95-569c-43a1-b631-e9991c57e11f","Type":"ContainerDied","Data":"38113f49b07a3c3d141ec77bf37e55645ff4fb0b8712ef144cb9798cf6f94ece"} Oct 30 22:47:18.092012 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.092009 4305 scope.go:122] "RemoveContainer" containerID="78dfb2e1111b58d26d99357b862751f16d0cd2af21f2c1f0069e21636e96ba34" Oct 30 22:47:18.093728 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.093700 4305 generic.go:356] "Generic (PLEG): container finished" podID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" containerID="02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0" exitCode=0 Oct 30 22:47:18.093897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.093748 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" Oct 30 22:47:18.093967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.093908 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" event={"ID":"9a550924-4a8a-4e4f-91a5-c3916688b6a1","Type":"ContainerDied","Data":"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0"} Oct 30 22:47:18.093967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.093943 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj" event={"ID":"9a550924-4a8a-4e4f-91a5-c3916688b6a1","Type":"ContainerDied","Data":"4f3a184b94b768b103f2bdfb3398fb22d2cd7e58121b09550e01147e229126da"} Oct 30 22:47:18.101840 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.101816 4305 scope.go:122] "RemoveContainer" containerID="02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0" Oct 30 22:47:18.111292 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.111250 4305 scope.go:122] "RemoveContainer" containerID="02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0" Oct 30 22:47:18.111599 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:18.111579 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0\": container with ID starting with 02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0 not found: ID does not exist" containerID="02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0" Oct 30 22:47:18.111677 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.111614 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0"} err="failed to get container status \"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0\": rpc error: code = NotFound desc = could not find container \"02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0\": container with ID starting with 02a53daf4755fac8eb3032bdc3bb96f82a11abfffef9950ed6bcd81aa15b40f0 not found: ID does not exist" Oct 30 22:47:18.251827 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.251781 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:47:18.257103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.257076 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-afb98c/test-model-afb98c-kserve-7bbccc7d9c-62fbj"] Oct 30 22:47:18.360402 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.360318 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:47:18.367366 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:18.367336 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d585d2-openshift-default-578cbd8b49-2qcxw"] Oct 30 22:47:19.517885 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:19.517854 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" path="/var/lib/kubelet/pods/9a550924-4a8a-4e4f-91a5-c3916688b6a1/volumes" Oct 30 22:47:19.518344 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:19.518329 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" path="/var/lib/kubelet/pods/d9f12a95-569c-43a1-b631-e9991c57e11f/volumes" Oct 30 22:47:32.446655 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.446595 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" containerID="cri-o://273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24" gracePeriod=45 Oct 30 22:47:32.740856 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.738547 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:47:32.830815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830764 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830835 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830857 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\" (UniqueName: \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830957 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.830986 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\") pod \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\" (UID: \"6b4b704b-1b8e-48f1-a515-bc7230b8fb89\") " Oct 30 22:47:32.831281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831011 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.831281 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831147 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.831371 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831284 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.831460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831442 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.831501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831467 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.831501 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.831482 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.833214 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.833182 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:32.833335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.833227 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.833335 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.833269 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws" pod "6b4b704b-1b8e-48f1-a515-bc7230b8fb89" (UID: "6b4b704b-1b8e-48f1-a515-bc7230b8fb89"). InnerVolumeSpecName "kube-api-access-q28ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:32.932447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.932416 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.932447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.932445 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28ws\" (UniqueName: \"kubernetes.io/projected/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-kube-api-access-q28ws\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.932447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:32.932454 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4b704b-1b8e-48f1-a515-bc7230b8fb89-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:33.177399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.177310 4305 generic.go:356] "Generic (PLEG): container finished" podID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerID="273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24" exitCode=0 Oct 30 22:47:33.177399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.177360 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" Oct 30 22:47:33.177399 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.177376 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" event={"ID":"6b4b704b-1b8e-48f1-a515-bc7230b8fb89","Type":"ContainerDied","Data":"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24"} Oct 30 22:47:33.177634 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.177416 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx" event={"ID":"6b4b704b-1b8e-48f1-a515-bc7230b8fb89","Type":"ContainerDied","Data":"f619399d16a36fc0fd7de67eac94791d8288f61c544bcd4bc7413f2a266e09d6"} Oct 30 22:47:33.177634 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.177438 4305 scope.go:122] "RemoveContainer" containerID="273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24" Oct 30 22:47:33.186680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.186661 4305 scope.go:122] "RemoveContainer" containerID="273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24" Oct 30 22:47:33.186960 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:33.186940 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24\": container with ID starting with 273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24 not found: ID does not exist" containerID="273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24" Oct 30 22:47:33.187017 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.186970 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24"} err="failed to get container status \"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24\": rpc error: code = NotFound desc = could not find container \"273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24\": container with ID starting with 273012ac7bd08671b0929dbe2f2368619aabbd0653fc2fafa7f350f6da967b24 not found: ID does not exist" Oct 30 22:47:33.337684 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.337651 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:47:33.383807 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.383763 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/distinct-w4-b13755-kserve-7d78b686b4-nc8kx"] Oct 30 22:47:33.517391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:33.517352 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" path="/var/lib/kubelet/pods/6b4b704b-1b8e-48f1-a515-bc7230b8fb89/volumes" Oct 30 22:47:46.109478 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:46.109440 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:47:53.958353 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.958171 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:47:53.958976 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.958480 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" containerID="cri-o://f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a" gracePeriod=30 Oct 30 22:47:53.991419 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.991386 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:47:53.991666 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.991615 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" containerID="cri-o://75656d8358dd70356a9aadc175ef5265a130d3637b3ff3c8d916c2c9f758f80d" gracePeriod=30 Oct 30 22:47:53.996410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.994890 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:47:53.996410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:53.995143 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" containerID="cri-o://e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90" gracePeriod=30 Oct 30 22:47:54.036094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:54.036062 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:47:54.036307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:54.036284 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" containerID="cri-o://8cb0fd44a3baf98049a6e706f89070aa8e983df94ebacd298678c9b3bf1fc168" gracePeriod=30 Oct 30 22:47:55.405730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:55.405693 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n42cb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:47:55.406126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:55.405759 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:47:56.567290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:56.567253 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zz7mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:47:56.567657 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:56.567323 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:47:58.442060 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.442024 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:47:58.442475 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.442211 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" podUID="088f3617-8a8e-490f-9bad-94bba1a95119" containerName="maas-api" containerID="cri-o://402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729" gracePeriod=30 Oct 30 22:47:58.733529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.733497 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:47:58.829282 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.829247 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\") pod \"088f3617-8a8e-490f-9bad-94bba1a95119\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " Oct 30 22:47:58.829591 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.829567 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\") pod \"088f3617-8a8e-490f-9bad-94bba1a95119\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " Oct 30 22:47:58.829745 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.829728 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") pod \"088f3617-8a8e-490f-9bad-94bba1a95119\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " Oct 30 22:47:58.829972 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.829953 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\" (UniqueName: \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\") pod \"088f3617-8a8e-490f-9bad-94bba1a95119\" (UID: \"088f3617-8a8e-490f-9bad-94bba1a95119\") " Oct 30 22:47:58.830218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.830193 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca" pod "088f3617-8a8e-490f-9bad-94bba1a95119" (UID: "088f3617-8a8e-490f-9bad-94bba1a95119"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:58.830613 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.830590 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/088f3617-8a8e-490f-9bad-94bba1a95119-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.834460 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.834427 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls" pod "088f3617-8a8e-490f-9bad-94bba1a95119" (UID: "088f3617-8a8e-490f-9bad-94bba1a95119"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:58.837227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.837165 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9" pod "088f3617-8a8e-490f-9bad-94bba1a95119" (UID: "088f3617-8a8e-490f-9bad-94bba1a95119"). InnerVolumeSpecName "kube-api-access-zdbb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:58.837350 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.837277 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs" pod "088f3617-8a8e-490f-9bad-94bba1a95119" (UID: "088f3617-8a8e-490f-9bad-94bba1a95119"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:58.933587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.933550 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.933587 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.933587 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/088f3617-8a8e-490f-9bad-94bba1a95119-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.933848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:58.933603 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdbb9\" (UniqueName: \"kubernetes.io/projected/088f3617-8a8e-490f-9bad-94bba1a95119-kube-api-access-zdbb9\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.032401 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:59.032365 4305 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6cec54_a4ca_4da9_8405_da222715a1e6.slice/crio-conmon-f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6cec54_a4ca_4da9_8405_da222715a1e6.slice/crio-f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:47:59.310686 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.310657 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:47:59.312339 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.312305 4305 generic.go:356] "Generic (PLEG): container finished" podID="088f3617-8a8e-490f-9bad-94bba1a95119" containerID="402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729" exitCode=0 Oct 30 22:47:59.312491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.312344 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" Oct 30 22:47:59.312491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.312384 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" event={"ID":"088f3617-8a8e-490f-9bad-94bba1a95119","Type":"ContainerDied","Data":"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729"} Oct 30 22:47:59.312491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.312439 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj" event={"ID":"088f3617-8a8e-490f-9bad-94bba1a95119","Type":"ContainerDied","Data":"770ead3d649b1a66a7da222e9a0b3f717a440039b249272c9bec4dcaf79c1bf0"} Oct 30 22:47:59.312491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.312468 4305 scope.go:122] "RemoveContainer" containerID="402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729" Oct 30 22:47:59.314781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.314622 4305 generic.go:356] "Generic (PLEG): container finished" podID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerID="f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a" exitCode=0 Oct 30 22:47:59.314781 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.314715 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" event={"ID":"4f6cec54-a4ca-4da9-8405-da222715a1e6","Type":"ContainerDied","Data":"f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a"} Oct 30 22:47:59.317039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.316896 4305 generic.go:356] "Generic (PLEG): container finished" podID="f0767789-10bd-4198-a644-3a7b441e4f03" containerID="75656d8358dd70356a9aadc175ef5265a130d3637b3ff3c8d916c2c9f758f80d" exitCode=0 Oct 30 22:47:59.317039 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.316976 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" event={"ID":"f0767789-10bd-4198-a644-3a7b441e4f03","Type":"ContainerDied","Data":"75656d8358dd70356a9aadc175ef5265a130d3637b3ff3c8d916c2c9f758f80d"} Oct 30 22:47:59.319238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.318869 4305 generic.go:356] "Generic (PLEG): container finished" podID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerID="e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90" exitCode=0 Oct 30 22:47:59.319238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.318963 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" event={"ID":"7a306129-d341-4737-b068-dd0b3c46aa1c","Type":"ContainerDied","Data":"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90"} Oct 30 22:47:59.319238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.319003 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" event={"ID":"7a306129-d341-4737-b068-dd0b3c46aa1c","Type":"ContainerDied","Data":"ed47f943010099a03aec5740a719a13c6e1d7d94a87b06b4c480ddeeb626d9bf"} Oct 30 22:47:59.319238 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.319080 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq" Oct 30 22:47:59.332930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.332481 4305 generic.go:356] "Generic (PLEG): container finished" podID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerID="8cb0fd44a3baf98049a6e706f89070aa8e983df94ebacd298678c9b3bf1fc168" exitCode=0 Oct 30 22:47:59.332930 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.332655 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" event={"ID":"db31495d-361f-46f5-a08a-c7b0b08fc24d","Type":"ContainerDied","Data":"8cb0fd44a3baf98049a6e706f89070aa8e983df94ebacd298678c9b3bf1fc168"} Oct 30 22:47:59.341897 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.341865 4305 scope.go:122] "RemoveContainer" containerID="402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729" Oct 30 22:47:59.342234 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:59.342204 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729\": container with ID starting with 402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729 not found: ID does not exist" containerID="402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729" Oct 30 22:47:59.342426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.342243 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729"} err="failed to get container status \"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729\": rpc error: code = NotFound desc = could not find container \"402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729\": container with ID starting with 402a38af3156e99e2e5d5204cdf084ea06601e8826c65c1a5ab80267848fb729 not found: ID does not exist" Oct 30 22:47:59.342426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.342268 4305 scope.go:122] "RemoveContainer" containerID="e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90" Oct 30 22:47:59.359360 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.359332 4305 scope.go:122] "RemoveContainer" containerID="e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90" Oct 30 22:47:59.359746 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:47:59.359718 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90\": container with ID starting with e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90 not found: ID does not exist" containerID="e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90" Oct 30 22:47:59.359900 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.359759 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90"} err="failed to get container status \"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90\": rpc error: code = NotFound desc = could not find container \"e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90\": container with ID starting with e8a1a03aa7726ee32a2e7d97bd19ecf21097e95c3de749563d68dc1b7de8ea90 not found: ID does not exist" Oct 30 22:47:59.383873 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.383775 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:47:59.386077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.386047 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:47:59.392605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.392584 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:47:59.440140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440103 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.440358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440168 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.440489 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440460 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.440618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440566 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.440618 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440627 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.440890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440659 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.440890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440695 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.440890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440740 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.440890 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440857 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.441145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440899 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440908 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:59.441145 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.440938 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441301 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441153 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.441301 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441275 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:59.441423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441313 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441423 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441403 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.441531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441448 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441531 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441478 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.441673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441532 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.441673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441579 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.441673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441599 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.441673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.441673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441656 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441687 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441740 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441789 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441879 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441908 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441959 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.441999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.441994 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\") pod \"7a306129-d341-4737-b068-dd0b3c46aa1c\" (UID: \"7a306129-d341-4737-b068-dd0b3c46aa1c\") " Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442023 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\") pod \"4f6cec54-a4ca-4da9-8405-da222715a1e6\" (UID: \"4f6cec54-a4ca-4da9-8405-da222715a1e6\") " Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442069 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\") pod \"db31495d-361f-46f5-a08a-c7b0b08fc24d\" (UID: \"db31495d-361f-46f5-a08a-c7b0b08fc24d\") " Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442469 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442483 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db31495d-361f-46f5-a08a-c7b0b08fc24d-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442493 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6cec54-a4ca-4da9-8405-da222715a1e6-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442502 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442512 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442521 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442647 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.442911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.442709 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.443513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.443073 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.443513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.443101 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:59.443513 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.443390 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.445038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.445014 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:59.445159 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.445122 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:59.445235 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.445153 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.445391 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.445358 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.445573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.445552 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.446154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.446128 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.447107 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.447049 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.447386 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.447361 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "kube-api-access-g45bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.447593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.447573 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.447817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.447702 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "kube-api-access-vk77p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.447817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.447746 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:59.448697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.448672 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token" pod "4f6cec54-a4ca-4da9-8405-da222715a1e6" (UID: "4f6cec54-a4ca-4da9-8405-da222715a1e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.448697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.448689 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.449511 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.449482 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "kube-api-access-jr8w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.449660 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.449638 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy" pod "7a306129-d341-4737-b068-dd0b3c46aa1c" (UID: "7a306129-d341-4737-b068-dd0b3c46aa1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.449935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.449910 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token" pod "db31495d-361f-46f5-a08a-c7b0b08fc24d" (UID: "db31495d-361f-46f5-a08a-c7b0b08fc24d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.536528 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.536495 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:47:59.539188 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.539161 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-afb98c-79697779b8-rgwnj"] Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543257 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543313 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543365 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543456 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543499 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543553 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543581 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543630 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.543660 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\") pod \"f0767789-10bd-4198-a644-3a7b441e4f03\" (UID: \"f0767789-10bd-4198-a644-3a7b441e4f03\") " Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544002 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544019 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544032 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544046 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a306129-d341-4737-b068-dd0b3c46aa1c-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544060 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544073 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544087 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544101 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544113 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544127 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8w6\" (UniqueName: \"kubernetes.io/projected/7a306129-d341-4737-b068-dd0b3c46aa1c-kube-api-access-jr8w6\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544144 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544158 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db31495d-361f-46f5-a08a-c7b0b08fc24d-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544171 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a306129-d341-4737-b068-dd0b3c46aa1c-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544184 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544197 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk77p\" (UniqueName: \"kubernetes.io/projected/4f6cec54-a4ca-4da9-8405-da222715a1e6-kube-api-access-vk77p\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544212 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45bf\" (UniqueName: \"kubernetes.io/projected/db31495d-361f-46f5-a08a-c7b0b08fc24d-kube-api-access-g45bf\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544227 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a306129-d341-4737-b068-dd0b3c46aa1c-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544240 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544254 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db31495d-361f-46f5-a08a-c7b0b08fc24d-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544267 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6cec54-a4ca-4da9-8405-da222715a1e6-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.544558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544280 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6cec54-a4ca-4da9-8405-da222715a1e6-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.545994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.545994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.544913 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.545994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.545243 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.545994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.545314 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:59.545994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.545406 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.547334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.547306 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.547430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.547381 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.547539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.547518 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "kube-api-access-jnxn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.547631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.547614 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo" pod "f0767789-10bd-4198-a644-3a7b441e4f03" (UID: "f0767789-10bd-4198-a644-3a7b441e4f03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:59.645307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645277 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxn6\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-kube-api-access-jnxn6\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645305 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0767789-10bd-4198-a644-3a7b441e4f03-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645307 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645314 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645323 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645332 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645341 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645349 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0767789-10bd-4198-a644-3a7b441e4f03-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645357 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0767789-10bd-4198-a644-3a7b441e4f03-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.645550 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:47:59.645366 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0767789-10bd-4198-a644-3a7b441e4f03-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:00.034222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.034184 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:48:00.040673 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.040394 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fj2bq"] Oct 30 22:48:00.340384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.340297 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" Oct 30 22:48:00.340384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.340302 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w" event={"ID":"4f6cec54-a4ca-4da9-8405-da222715a1e6","Type":"ContainerDied","Data":"817d5a8f11d2c4cda0d22589434ea5423cf551d2d085d901e3125a2211829989"} Oct 30 22:48:00.340384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.340369 4305 scope.go:122] "RemoveContainer" containerID="f357ac8f03523223172f73e40a68c56c615ebf383dc4564c817233dc1d28943a" Oct 30 22:48:00.341869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.341835 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" event={"ID":"f0767789-10bd-4198-a644-3a7b441e4f03","Type":"ContainerDied","Data":"0d396f57a8d5fd2b312a754973491fd95ab163c42a75d2d964561ace1c9ef2fc"} Oct 30 22:48:00.341869 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.341857 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb" Oct 30 22:48:00.346440 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.346406 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" event={"ID":"db31495d-361f-46f5-a08a-c7b0b08fc24d","Type":"ContainerDied","Data":"60c7717c78f9b1788bd44fdb53ddc63a7d45c754e62db0d1dc7123a17f9d58e8"} Oct 30 22:48:00.346605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.346586 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk" Oct 30 22:48:00.360193 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.360159 4305 scope.go:122] "RemoveContainer" containerID="75656d8358dd70356a9aadc175ef5265a130d3637b3ff3c8d916c2c9f758f80d" Oct 30 22:48:00.369844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.369818 4305 scope.go:122] "RemoveContainer" containerID="8cb0fd44a3baf98049a6e706f89070aa8e983df94ebacd298678c9b3bf1fc168" Oct 30 22:48:00.495970 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.495934 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:48:00.501001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.500972 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zz7mk"] Oct 30 22:48:00.605773 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.605684 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:48:00.614830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.614787 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n42cb"] Oct 30 22:48:00.719608 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.719573 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:48:00.725229 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:00.725200 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lhj8w"] Oct 30 22:48:01.517290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:01.517258 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088f3617-8a8e-490f-9bad-94bba1a95119" path="/var/lib/kubelet/pods/088f3617-8a8e-490f-9bad-94bba1a95119/volumes" Oct 30 22:48:01.517924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:01.517904 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" path="/var/lib/kubelet/pods/4f6cec54-a4ca-4da9-8405-da222715a1e6/volumes" Oct 30 22:48:01.518352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:01.518332 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" path="/var/lib/kubelet/pods/7a306129-d341-4737-b068-dd0b3c46aa1c/volumes" Oct 30 22:48:01.519120 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:01.519100 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" path="/var/lib/kubelet/pods/db31495d-361f-46f5-a08a-c7b0b08fc24d/volumes" Oct 30 22:48:01.519545 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:01.519527 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" path="/var/lib/kubelet/pods/f0767789-10bd-4198-a644-3a7b441e4f03/volumes" Oct 30 22:48:16.109058 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:48:16.109021 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:48:36.616394 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.616307 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:48:36.616866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.616531 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" podUID="112fece8-9f56-4481-a69e-6bc3b47baf0d" containerName="maas-api" containerID="cri-o://d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b" gracePeriod=30 Oct 30 22:48:36.865785 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.865755 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:48:36.918881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.918771 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\") pod \"112fece8-9f56-4481-a69e-6bc3b47baf0d\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " Oct 30 22:48:36.918881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.918882 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\") pod \"112fece8-9f56-4481-a69e-6bc3b47baf0d\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " Oct 30 22:48:36.919137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.918948 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\" (UniqueName: \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\") pod \"112fece8-9f56-4481-a69e-6bc3b47baf0d\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " Oct 30 22:48:36.919137 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.919004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") pod \"112fece8-9f56-4481-a69e-6bc3b47baf0d\" (UID: \"112fece8-9f56-4481-a69e-6bc3b47baf0d\") " Oct 30 22:48:36.919320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.919292 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca" pod "112fece8-9f56-4481-a69e-6bc3b47baf0d" (UID: "112fece8-9f56-4481-a69e-6bc3b47baf0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:36.921057 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.921019 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk" pod "112fece8-9f56-4481-a69e-6bc3b47baf0d" (UID: "112fece8-9f56-4481-a69e-6bc3b47baf0d"). InnerVolumeSpecName "kube-api-access-w9vpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:36.921179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.921075 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs" pod "112fece8-9f56-4481-a69e-6bc3b47baf0d" (UID: "112fece8-9f56-4481-a69e-6bc3b47baf0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:36.921179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:36.921098 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls" pod "112fece8-9f56-4481-a69e-6bc3b47baf0d" (UID: "112fece8-9f56-4481-a69e-6bc3b47baf0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:37.020568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.020529 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:37.020568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.020561 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112fece8-9f56-4481-a69e-6bc3b47baf0d-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:37.020568 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.020572 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112fece8-9f56-4481-a69e-6bc3b47baf0d-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:37.020819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.020582 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9vpk\" (UniqueName: \"kubernetes.io/projected/112fece8-9f56-4481-a69e-6bc3b47baf0d-kube-api-access-w9vpk\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:37.544392 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.544356 4305 generic.go:356] "Generic (PLEG): container finished" podID="112fece8-9f56-4481-a69e-6bc3b47baf0d" containerID="d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b" exitCode=0 Oct 30 22:48:37.544535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.544402 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" Oct 30 22:48:37.544535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.544437 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" event={"ID":"112fece8-9f56-4481-a69e-6bc3b47baf0d","Type":"ContainerDied","Data":"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b"} Oct 30 22:48:37.544535 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.544511 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt" event={"ID":"112fece8-9f56-4481-a69e-6bc3b47baf0d","Type":"ContainerDied","Data":"51b0fab3c5dce2611a7e1ed062c954442085b8cd161aba08572a87ea45ccb3a6"} Oct 30 22:48:37.544792 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.544546 4305 scope.go:122] "RemoveContainer" containerID="d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b" Oct 30 22:48:37.556073 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.556054 4305 scope.go:122] "RemoveContainer" containerID="d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b" Oct 30 22:48:37.556453 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:48:37.556427 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b\": container with ID starting with d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b not found: ID does not exist" containerID="d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b" Oct 30 22:48:37.556532 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.556468 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b"} err="failed to get container status \"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b\": rpc error: code = NotFound desc = could not find container \"d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b\": container with ID starting with d0e50e3daf5eb6f9e0bd56ec843c5833eddddd1c1d8d79270653bee7e377c92b not found: ID does not exist" Oct 30 22:48:37.700778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.700742 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:48:37.746768 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:37.746734 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-926fc6-567598bdc6-47dpt"] Oct 30 22:48:39.516916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:39.516885 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112fece8-9f56-4481-a69e-6bc3b47baf0d" path="/var/lib/kubelet/pods/112fece8-9f56-4481-a69e-6bc3b47baf0d/volumes" Oct 30 22:48:46.019316 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:46.019283 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:48:46.019849 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:46.019469 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" containerID="cri-o://7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1" gracePeriod=30 Oct 30 22:48:46.110261 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:48:46.110214 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:48:47.434375 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:47.434335 4305 patch_prober.go:29] interesting pod/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:48:47.434775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:47.434408 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:48:51.271479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.271454 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:48:51.362146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362053 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362093 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362205 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362236 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362272 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362326 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362396 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362361 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362413 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362445 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\") pod \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\" (UID: \"50199d00-3b3e-44ff-b6d2-f9f6b5830255\") " Oct 30 22:48:51.362670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362593 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:51.362670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362626 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:51.362670 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362662 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:51.362995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362854 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.362995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362876 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.362995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362901 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.362995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362872 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:51.362995 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.362946 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:51.364468 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.364441 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:48:51.364593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.364477 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:51.364645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.364601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:51.364645 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.364621 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd" pod "50199d00-3b3e-44ff-b6d2-f9f6b5830255" (UID: "50199d00-3b3e-44ff-b6d2-f9f6b5830255"). InnerVolumeSpecName "kube-api-access-zz2cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:51.463842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463811 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.463994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463854 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.463994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463869 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.463994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463880 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50199d00-3b3e-44ff-b6d2-f9f6b5830255-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.463994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463890 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2cd\" (UniqueName: \"kubernetes.io/projected/50199d00-3b3e-44ff-b6d2-f9f6b5830255-kube-api-access-zz2cd\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.463994 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.463900 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50199d00-3b3e-44ff-b6d2-f9f6b5830255-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:51.611252 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.611219 4305 generic.go:356] "Generic (PLEG): container finished" podID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerID="7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1" exitCode=0 Oct 30 22:48:51.611412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.611267 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" Oct 30 22:48:51.611412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.611279 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerDied","Data":"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1"} Oct 30 22:48:51.611412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.611330 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7" event={"ID":"50199d00-3b3e-44ff-b6d2-f9f6b5830255","Type":"ContainerDied","Data":"6cbec83833fb2f7e275ecd0798a290595cb76f10b3d9601ce127f43713c6cd70"} Oct 30 22:48:51.611412 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.611353 4305 scope.go:122] "RemoveContainer" containerID="7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1" Oct 30 22:48:51.620124 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.620104 4305 scope.go:122] "RemoveContainer" containerID="0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba" Oct 30 22:48:51.627911 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.627889 4305 scope.go:122] "RemoveContainer" containerID="7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1" Oct 30 22:48:51.628166 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:48:51.628145 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1\": container with ID starting with 7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1 not found: ID does not exist" containerID="7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1" Oct 30 22:48:51.628218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.628179 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1"} err="failed to get container status \"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1\": rpc error: code = NotFound desc = could not find container \"7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1\": container with ID starting with 7737faef2495af71e42b32c89fa320c4d2b4fabe80cde3a37c442af9cb01b1b1 not found: ID does not exist" Oct 30 22:48:51.628218 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.628199 4305 scope.go:122] "RemoveContainer" containerID="0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba" Oct 30 22:48:51.628452 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:48:51.628433 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba\": container with ID starting with 0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba not found: ID does not exist" containerID="0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba" Oct 30 22:48:51.628503 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.628458 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba"} err="failed to get container status \"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba\": rpc error: code = NotFound desc = could not find container \"0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba\": container with ID starting with 0858b3a146d7c88f9286ba0d55dd8e96fae466fc93e816a5ed49eedd060835ba not found: ID does not exist" Oct 30 22:48:51.765291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.765251 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:48:51.771818 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:51.771778 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-926fc6-openshift-default-6b97494567-znrf7"] Oct 30 22:48:53.516914 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:48:53.516880 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" path="/var/lib/kubelet/pods/50199d00-3b3e-44ff-b6d2-f9f6b5830255/volumes" Oct 30 22:49:05.662839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.662788 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:49:05.663294 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.663044 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerName="maas-api" containerID="cri-o://455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620" gracePeriod=30 Oct 30 22:49:05.695100 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.695067 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.94:8443/health\": dial tcp 10.134.0.94:8443: connect: connection refused" Oct 30 22:49:05.925571 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.925514 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:49:05.999000 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.998970 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\") pod \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " Oct 30 22:49:05.999154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.999041 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") pod \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " Oct 30 22:49:05.999209 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.999177 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\") pod \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " Oct 30 22:49:05.999251 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.999218 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\" (UniqueName: \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\") pod \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\" (UID: \"6f5d36d0-8ca4-4e96-b3c3-6b9648996932\") " Oct 30 22:49:05.999428 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.999404 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca" pod "6f5d36d0-8ca4-4e96-b3c3-6b9648996932" (UID: "6f5d36d0-8ca4-4e96-b3c3-6b9648996932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:05.999522 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:05.999508 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.001196 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.001167 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls" pod "6f5d36d0-8ca4-4e96-b3c3-6b9648996932" (UID: "6f5d36d0-8ca4-4e96-b3c3-6b9648996932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:06.001333 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.001211 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs" pod "6f5d36d0-8ca4-4e96-b3c3-6b9648996932" (UID: "6f5d36d0-8ca4-4e96-b3c3-6b9648996932"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:06.001333 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.001271 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t" pod "6f5d36d0-8ca4-4e96-b3c3-6b9648996932" (UID: "6f5d36d0-8ca4-4e96-b3c3-6b9648996932"). InnerVolumeSpecName "kube-api-access-7hp6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:06.100256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.100217 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.100256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.100249 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.100256 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.100262 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hp6t\" (UniqueName: \"kubernetes.io/projected/6f5d36d0-8ca4-4e96-b3c3-6b9648996932-kube-api-access-7hp6t\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:06.689719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.689686 4305 generic.go:356] "Generic (PLEG): container finished" podID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerID="455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620" exitCode=0 Oct 30 22:49:06.690242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.689733 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" Oct 30 22:49:06.690242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.689771 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" event={"ID":"6f5d36d0-8ca4-4e96-b3c3-6b9648996932","Type":"ContainerDied","Data":"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620"} Oct 30 22:49:06.690242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.689843 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss" event={"ID":"6f5d36d0-8ca4-4e96-b3c3-6b9648996932","Type":"ContainerDied","Data":"3197a4c65e7c318a9bfaaa0f13bc5b7e4d3ab3d3a83bff8ddf69664cb2f0a2ee"} Oct 30 22:49:06.690242 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.689868 4305 scope.go:122] "RemoveContainer" containerID="455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620" Oct 30 22:49:06.703253 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.703230 4305 scope.go:122] "RemoveContainer" containerID="455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620" Oct 30 22:49:06.703632 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:06.703603 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620\": container with ID starting with 455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620 not found: ID does not exist" containerID="455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620" Oct 30 22:49:06.703732 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.703641 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620"} err="failed to get container status \"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620\": rpc error: code = NotFound desc = could not find container \"455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620\": container with ID starting with 455584bd83af0e6cda175823ecb35404b5137afba4b68573fa9909fbd26ff620 not found: ID does not exist" Oct 30 22:49:06.844346 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.844313 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:49:06.890508 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:06.890468 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5c2f68-d67b6f789-v5tss"] Oct 30 22:49:07.520459 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:07.520420 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" path="/var/lib/kubelet/pods/6f5d36d0-8ca4-4e96-b3c3-6b9648996932/volumes" Oct 30 22:49:16.108731 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:16.108694 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:49:20.296991 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:20.296955 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:49:20.342947 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:20.342916 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:49:35.318009 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.317951 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" containerID="cri-o://b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566" gracePeriod=45 Oct 30 22:49:35.363652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.363599 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" containerID="cri-o://df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af" gracePeriod=45 Oct 30 22:49:35.430605 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.430570 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused" Oct 30 22:49:35.606162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.606137 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:49:35.631861 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.631837 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:49:35.787775 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.787746 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\" (UniqueName: \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.787993 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.787841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.787992 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\" (UniqueName: \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788046 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.788186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788077 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.788186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788083 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788186 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788160 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.788347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788192 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788283 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788292 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.788347 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788322 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\") pod \"4bb180a4-1816-40e6-90b8-631bddf90c2d\" (UID: \"4bb180a4-1816-40e6-90b8-631bddf90c2d\") " Oct 30 22:49:35.788557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788391 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.788557 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788450 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.788662 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788644 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.788722 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788648 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.788722 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788689 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\") pod \"f198148e-24d1-4355-93b8-c48af59929a9\" (UID: \"f198148e-24d1-4355-93b8-c48af59929a9\") " Oct 30 22:49:35.788860 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788688 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.788922 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.788887 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789254 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789281 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789295 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789308 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789321 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.789417 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.789334 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.790539 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790500 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.790680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790622 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "kube-api-access-zpnm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.790680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790659 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "kube-api-access-q5swb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.790842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790818 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs" pod "4bb180a4-1816-40e6-90b8-631bddf90c2d" (UID: "4bb180a4-1816-40e6-90b8-631bddf90c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.790936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790911 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.790974 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.790951 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs" pod "f198148e-24d1-4355-93b8-c48af59929a9" (UID: "f198148e-24d1-4355-93b8-c48af59929a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.828676 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.828640 4305 generic.go:356] "Generic (PLEG): container finished" podID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerID="b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566" exitCode=0 Oct 30 22:49:35.828859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.828675 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" event={"ID":"4bb180a4-1816-40e6-90b8-631bddf90c2d","Type":"ContainerDied","Data":"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566"} Oct 30 22:49:35.828859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.828734 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" event={"ID":"4bb180a4-1816-40e6-90b8-631bddf90c2d","Type":"ContainerDied","Data":"ef835c1978c62be6a2a105233a96669e884a73812c4a2f3247480e2f5c60c1c1"} Oct 30 22:49:35.828859 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.828758 4305 scope.go:122] "RemoveContainer" containerID="b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566" Oct 30 22:49:35.829836 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.828910 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd" Oct 30 22:49:35.830759 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.830734 4305 generic.go:356] "Generic (PLEG): container finished" podID="f198148e-24d1-4355-93b8-c48af59929a9" containerID="df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af" exitCode=0 Oct 30 22:49:35.830951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.830770 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" event={"ID":"f198148e-24d1-4355-93b8-c48af59929a9","Type":"ContainerDied","Data":"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af"} Oct 30 22:49:35.830951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.830821 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" event={"ID":"f198148e-24d1-4355-93b8-c48af59929a9","Type":"ContainerDied","Data":"7170209524d7a3b2e79003624ded71018f2ecaca12e0004d8d53001b5cdd3c1f"} Oct 30 22:49:35.830951 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.830835 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz" Oct 30 22:49:35.847320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.847295 4305 scope.go:122] "RemoveContainer" containerID="b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566" Oct 30 22:49:35.847639 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:35.847618 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566\": container with ID starting with b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566 not found: ID does not exist" containerID="b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566" Oct 30 22:49:35.847717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.847647 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566"} err="failed to get container status \"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566\": rpc error: code = NotFound desc = could not find container \"b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566\": container with ID starting with b395e9a73c57743a8eaf07fc6389e6fca5a76c851e18975e64dcbb1b1e0d6566 not found: ID does not exist" Oct 30 22:49:35.847717 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.847663 4305 scope.go:122] "RemoveContainer" containerID="df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af" Oct 30 22:49:35.857168 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.857149 4305 scope.go:122] "RemoveContainer" containerID="df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af" Oct 30 22:49:35.857473 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:35.857452 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af\": container with ID starting with df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af not found: ID does not exist" containerID="df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af" Oct 30 22:49:35.857549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.857478 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af"} err="failed to get container status \"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af\": rpc error: code = NotFound desc = could not find container \"df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af\": container with ID starting with df5bea02dc8a0d85f9314cb0b98880fc87a5b785551e03697641812bd3e6b9af not found: ID does not exist" Oct 30 22:49:35.890675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890641 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb180a4-1816-40e6-90b8-631bddf90c2d-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.890675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890669 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f198148e-24d1-4355-93b8-c48af59929a9-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.890881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890683 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnm4\" (UniqueName: \"kubernetes.io/projected/f198148e-24d1-4355-93b8-c48af59929a9-kube-api-access-zpnm4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.890881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890700 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5swb\" (UniqueName: \"kubernetes.io/projected/4bb180a4-1816-40e6-90b8-631bddf90c2d-kube-api-access-q5swb\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.890881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890712 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f198148e-24d1-4355-93b8-c48af59929a9-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.890881 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:35.890726 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb180a4-1816-40e6-90b8-631bddf90c2d-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:36.038842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:36.035559 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:49:36.041022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:36.040995 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/premium-w2-eb4aa1-kserve-85db4d47bb-skxpz"] Oct 30 22:49:36.141542 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:36.141471 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:49:36.148022 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:36.148001 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-w2-eb4aa1-kserve-748c95fb49-rpltd"] Oct 30 22:49:37.517163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:37.517132 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" path="/var/lib/kubelet/pods/4bb180a4-1816-40e6-90b8-631bddf90c2d/volumes" Oct 30 22:49:37.517671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:37.517653 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f198148e-24d1-4355-93b8-c48af59929a9" path="/var/lib/kubelet/pods/f198148e-24d1-4355-93b8-c48af59929a9/volumes" Oct 30 22:49:46.109005 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:46.108964 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:49:50.607289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.607250 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:49:50.607821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.607457 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" podUID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" containerName="maas-api" containerID="cri-o://c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f" gracePeriod=30 Oct 30 22:49:50.862075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.862022 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:49:50.901729 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.901692 4305 generic.go:356] "Generic (PLEG): container finished" podID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" containerID="c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f" exitCode=0 Oct 30 22:49:50.901906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.901749 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" Oct 30 22:49:50.901906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.901777 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" event={"ID":"a0d51899-dda4-46b8-8cba-fa6aa75279a6","Type":"ContainerDied","Data":"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f"} Oct 30 22:49:50.901906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.901847 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5" event={"ID":"a0d51899-dda4-46b8-8cba-fa6aa75279a6","Type":"ContainerDied","Data":"dbb4ff4c316846729c0ffda3f2030924e2166b323132a2d6c4a6f798fb1f1025"} Oct 30 22:49:50.901906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.901871 4305 scope.go:122] "RemoveContainer" containerID="c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f" Oct 30 22:49:50.910680 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.910661 4305 scope.go:122] "RemoveContainer" containerID="c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f" Oct 30 22:49:50.911046 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:49:50.911023 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f\": container with ID starting with c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f not found: ID does not exist" containerID="c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f" Oct 30 22:49:50.911143 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.911056 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f"} err="failed to get container status \"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f\": rpc error: code = NotFound desc = could not find container \"c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f\": container with ID starting with c9b2e2572af6853690203bf3fff92dc9a355a4d150c1efd03e0785f9fb88a57f not found: ID does not exist" Oct 30 22:49:50.920829 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.920786 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") pod \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " Oct 30 22:49:50.920935 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.920907 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\") pod \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " Oct 30 22:49:50.920997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.920951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\" (UniqueName: \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\") pod \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " Oct 30 22:49:50.920997 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.920993 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\") pod \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\" (UID: \"a0d51899-dda4-46b8-8cba-fa6aa75279a6\") " Oct 30 22:49:50.921330 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.921304 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca" pod "a0d51899-dda4-46b8-8cba-fa6aa75279a6" (UID: "a0d51899-dda4-46b8-8cba-fa6aa75279a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:50.922902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.922866 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls" pod "a0d51899-dda4-46b8-8cba-fa6aa75279a6" (UID: "a0d51899-dda4-46b8-8cba-fa6aa75279a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:50.923076 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.923055 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs" pod "a0d51899-dda4-46b8-8cba-fa6aa75279a6" (UID: "a0d51899-dda4-46b8-8cba-fa6aa75279a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:50.923276 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:50.923257 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4" pod "a0d51899-dda4-46b8-8cba-fa6aa75279a6" (UID: "a0d51899-dda4-46b8-8cba-fa6aa75279a6"). InnerVolumeSpecName "kube-api-access-fftp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:51.021734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.021703 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fftp4\" (UniqueName: \"kubernetes.io/projected/a0d51899-dda4-46b8-8cba-fa6aa75279a6-kube-api-access-fftp4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:51.021734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.021729 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:51.021734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.021741 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d51899-dda4-46b8-8cba-fa6aa75279a6-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:51.021971 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.021750 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d51899-dda4-46b8-8cba-fa6aa75279a6-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:51.315591 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.315558 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:49:51.320190 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.320166 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eb4aa1-757bfd789c-rl6h5"] Oct 30 22:49:51.516916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:49:51.516842 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" path="/var/lib/kubelet/pods/a0d51899-dda4-46b8-8cba-fa6aa75279a6/volumes" Oct 30 22:50:06.235334 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.235241 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:50:06.235790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.235438 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8cf787766-jpmcd" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" containerID="cri-o://444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9" gracePeriod=30 Oct 30 22:50:06.472617 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.472593 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:50:06.573707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.573615 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\") pod \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " Oct 30 22:50:06.573707 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.573688 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\" (UniqueName: \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\") pod \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " Oct 30 22:50:06.574072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.573753 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\") pod \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\" (UID: \"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734\") " Oct 30 22:50:06.575958 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.575928 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs" pod "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" (UID: "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:06.576070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.575956 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5" pod "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" (UID: "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734"). InnerVolumeSpecName "kube-api-access-qtfm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:06.576070 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.576001 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert" pod "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" (UID: "53f3ca0f-c4e6-4d19-8e70-1f653e0b8734"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:06.675325 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.675291 4305 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-webhook-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:06.675325 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.675322 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:06.675519 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.675334 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfm5\" (UniqueName: \"kubernetes.io/projected/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734-kube-api-access-qtfm5\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:06.972247 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.972216 4305 generic.go:356] "Generic (PLEG): container finished" podID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerID="444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9" exitCode=0 Oct 30 22:50:06.972430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.972279 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-jpmcd" Oct 30 22:50:06.972430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.972304 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerDied","Data":"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9"} Oct 30 22:50:06.972430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.972362 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-jpmcd" event={"ID":"53f3ca0f-c4e6-4d19-8e70-1f653e0b8734","Type":"ContainerDied","Data":"34266fb778ec26c1498ceafbf743d4b438db75ab7d86509e8a3d0d9f797f13b1"} Oct 30 22:50:06.972430 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.972383 4305 scope.go:122] "RemoveContainer" containerID="444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9" Oct 30 22:50:06.982919 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.982897 4305 scope.go:122] "RemoveContainer" containerID="60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786" Oct 30 22:50:06.990688 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.990669 4305 scope.go:122] "RemoveContainer" containerID="444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9" Oct 30 22:50:06.990988 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:50:06.990968 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9\": container with ID starting with 444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9 not found: ID does not exist" containerID="444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9" Oct 30 22:50:06.991072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.990999 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9"} err="failed to get container status \"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9\": rpc error: code = NotFound desc = could not find container \"444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9\": container with ID starting with 444e619a2b163a96985b74fa2a35e90fdb340e32c421f33a2e33f1dda11eb9a9 not found: ID does not exist" Oct 30 22:50:06.991072 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.991017 4305 scope.go:122] "RemoveContainer" containerID="60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786" Oct 30 22:50:06.991269 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:50:06.991251 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786\": container with ID starting with 60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786 not found: ID does not exist" containerID="60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786" Oct 30 22:50:06.991310 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:06.991276 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786"} err="failed to get container status \"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786\": rpc error: code = NotFound desc = could not find container \"60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786\": container with ID starting with 60749a7fdba9182c8426012e71ff9e9aa5a80714017ac00390c8f5119d91d786 not found: ID does not exist" Oct 30 22:50:07.126135 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:07.126101 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:50:07.171982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:07.171951 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8cf787766-jpmcd"] Oct 30 22:50:07.516984 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:07.516950 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" path="/var/lib/kubelet/pods/53f3ca0f-c4e6-4d19-8e70-1f653e0b8734/volumes" Oct 30 22:50:08.033389 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033348 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:08.033709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033697 4305 cpu_manager.go:414] "containerMap: removing container" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033710 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033719 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0e95a88c-18da-4a65-b471-20528f5a7cc6" containerName="maas-api" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033725 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e95a88c-18da-4a65-b471-20528f5a7cc6" containerName="maas-api" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033735 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033740 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" Oct 30 22:50:08.033754 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033753 4305 cpu_manager.go:414] "containerMap: removing container" podUID="cc77f4d4-5742-45c6-a838-c316551364f5" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033759 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc77f4d4-5742-45c6-a838-c316551364f5" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033769 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033773 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033779 4305 cpu_manager.go:414] "containerMap: removing container" podUID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033784 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033792 4305 cpu_manager.go:414] "containerMap: removing container" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033811 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033821 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033828 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033838 4305 cpu_manager.go:414] "containerMap: removing container" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033844 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033854 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033860 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033873 4305 cpu_manager.go:414] "containerMap: removing container" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033877 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033883 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033887 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033894 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033898 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033904 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033909 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033914 4305 cpu_manager.go:414] "containerMap: removing container" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033918 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033927 4305 cpu_manager.go:414] "containerMap: removing container" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033933 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033940 4305 cpu_manager.go:414] "containerMap: removing container" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033944 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033950 4305 cpu_manager.go:414] "containerMap: removing container" podUID="112fece8-9f56-4481-a69e-6bc3b47baf0d" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033954 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112fece8-9f56-4481-a69e-6bc3b47baf0d" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033961 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033966 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033972 4305 cpu_manager.go:414] "containerMap: removing container" podUID="088f3617-8a8e-490f-9bad-94bba1a95119" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033977 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088f3617-8a8e-490f-9bad-94bba1a95119" containerName="maas-api" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033984 4305 cpu_manager.go:414] "containerMap: removing container" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033988 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.033982 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.033998 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034002 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034010 4305 cpu_manager.go:414] "containerMap: removing container" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034015 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034022 4305 cpu_manager.go:414] "containerMap: removing container" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034026 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034035 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8ad7b205-65c7-4f23-8033-64dc828dda1b" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034040 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad7b205-65c7-4f23-8033-64dc828dda1b" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034045 4305 cpu_manager.go:414] "containerMap: removing container" podUID="be96707f-c6f0-4313-9543-fd586492afa5" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034049 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be96707f-c6f0-4313-9543-fd586492afa5" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034057 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034061 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034068 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034072 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034128 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0767789-10bd-4198-a644-3a7b441e4f03" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034138 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc77f4d4-5742-45c6-a838-c316551364f5" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034146 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034151 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a550924-4a8a-4e4f-91a5-c3916688b6a1" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034158 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034163 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5d36d0-8ca4-4e96-b3c3-6b9648996932" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034169 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034175 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034180 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3d1b90-8506-493c-bc2a-3e79d21bba50" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034185 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb180a4-1816-40e6-90b8-631bddf90c2d" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034190 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6cec54-a4ca-4da9-8405-da222715a1e6" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034196 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a306129-d341-4737-b068-dd0b3c46aa1c" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034200 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="998636d1-6870-4142-becc-d4b6c74795ce" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034206 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="db31495d-361f-46f5-a08a-c7b0b08fc24d" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034212 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034217 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034221 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="112fece8-9f56-4481-a69e-6bc3b47baf0d" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034226 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f198148e-24d1-4355-93b8-c48af59929a9" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034231 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="088f3617-8a8e-490f-9bad-94bba1a95119" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034235 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f3ca0f-c4e6-4d19-8e70-1f653e0b8734" containerName="manager" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034241 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4b704b-1b8e-48f1-a515-bc7230b8fb89" containerName="main" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034246 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034251 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="50199d00-3b3e-44ff-b6d2-f9f6b5830255" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034257 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad7b205-65c7-4f23-8033-64dc828dda1b" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034262 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="be96707f-c6f0-4313-9543-fd586492afa5" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034267 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d51899-dda4-46b8-8cba-fa6aa75279a6" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034272 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6f6d42-621c-4f0e-b194-118512ee0366" containerName="maas-api" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034278 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="324da313-68f4-4270-9984-91d39e6c15c6" containerName="istio-proxy" Oct 30 22:50:08.035117 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.034282 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e95a88c-18da-4a65-b471-20528f5a7cc6" containerName="maas-api" Oct 30 22:50:08.038926 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.038903 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.135452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.135400 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:50:08.135452 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.135427 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:50:08.135678 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.135483 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ct9hv\"" Oct 30 22:50:08.136581 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.136563 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:50:08.137817 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.137784 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:08.188577 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.188532 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.188760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.188597 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgqc\" (UniqueName: \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.188760 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.188736 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.289834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.289702 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.289834 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.289766 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.290045 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.289842 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgqc\" (UniqueName: \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.292266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.292238 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.292388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.292265 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.348377 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.348343 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgqc\" (UniqueName: \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\") pod \"maas-controller-8cf787766-tt8cp\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.359140 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.359112 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:08.733565 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.733524 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:08.740218 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:50:08.740190 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff25959_2b8f_49f0_b1b0_db5dd688f410.slice/crio-08c8e2bd3284a7900fd7cc1357d1de89db3875166f81ea32276e01e9b24b072e WatchSource:0}: Error finding container 08c8e2bd3284a7900fd7cc1357d1de89db3875166f81ea32276e01e9b24b072e: Status 404 returned error can't find the container with id 08c8e2bd3284a7900fd7cc1357d1de89db3875166f81ea32276e01e9b24b072e Oct 30 22:50:08.741538 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.741519 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:50:08.984149 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:08.984059 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-tt8cp" event={"ID":"7ff25959-2b8f-49f0-b1b0-db5dd688f410","Type":"ContainerStarted","Data":"08c8e2bd3284a7900fd7cc1357d1de89db3875166f81ea32276e01e9b24b072e"} Oct 30 22:50:09.990329 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:09.990240 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-tt8cp" event={"ID":"7ff25959-2b8f-49f0-b1b0-db5dd688f410","Type":"ContainerStarted","Data":"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b"} Oct 30 22:50:09.990948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:09.990358 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:10.102007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:10.101953 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8cf787766-tt8cp" podStartSLOduration=2.148184675 podStartE2EDuration="3.101937831s" totalImagesPullingTime="953.753156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:10.100254603 +0000 UTC m=+2930.695342935" watchObservedRunningTime="2026-10-30 22:50:10.101937831 +0000 UTC m=+2930.697026136" Oct 30 22:50:16.108576 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:50:16.108536 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:50:20.997125 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:20.997092 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:40.238299 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238266 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5"] Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238687 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238702 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238714 4305 cpu_manager.go:414] "containerMap: removing container" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238720 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cede09-a059-4eda-9c3c-f7f45097a47e" containerName="istio-proxy" Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238729 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:40.238784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238734 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:40.239006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238817 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f12a95-569c-43a1-b631-e9991c57e11f" containerName="istio-proxy" Oct 30 22:50:40.239006 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.238827 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48eddfb-85f8-4fc4-aa5a-44dff6778940" containerName="istio-proxy" Oct 30 22:50:40.243784 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.243762 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.278842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.278781 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.278977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.278882 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.278977 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.278913 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.278974 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279094 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.279074 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279134 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.279106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.279175 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279263 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.279249 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.279300 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.279282 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbh6\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-kube-api-access-wjbh6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.284640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.284617 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw"] Oct 30 22:50:40.288020 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.288002 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.304065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.304049 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\"" Oct 30 22:50:40.330606 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.330571 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5"] Oct 30 22:50:40.353931 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.353907 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw"] Oct 30 22:50:40.379778 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379751 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.379945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379786 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbh6\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-kube-api-access-wjbh6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.379945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379823 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982a22ec-5768-4a3c-843d-42d811121213-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.379945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379860 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.379945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379883 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.379945 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379913 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.379967 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380040 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380071 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380125 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380151 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380180 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380221 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380249 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380277 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380320 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380439 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380489 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvrn\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-kube-api-access-5lvrn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380593 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380521 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982a22ec-5768-4a3c-843d-42d811121213-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.380899 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.380609 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.381028 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.381011 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.382623 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.382599 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.382983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.382962 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.437494 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.437460 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.437625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.437532 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbh6\" (UniqueName: \"kubernetes.io/projected/a1ffecbd-ee8d-4e9f-a7bf-f4348f280830-kube-api-access-wjbh6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cg4z5\" (UID: \"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.481194 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481166 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481206 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481245 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481285 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481341 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvrn\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-kube-api-access-5lvrn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481366 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982a22ec-5768-4a3c-843d-42d811121213-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481453 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982a22ec-5768-4a3c-843d-42d811121213-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481501 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481589 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481530 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481698 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481790 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481735 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481865 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.481989 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.481959 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.482144 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.482122 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982a22ec-5768-4a3c-843d-42d811121213-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.483907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.483884 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982a22ec-5768-4a3c-843d-42d811121213-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.484044 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.484024 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982a22ec-5768-4a3c-843d-42d811121213-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.538835 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.538738 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.539019 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.538999 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvrn\" (UniqueName: \"kubernetes.io/projected/982a22ec-5768-4a3c-843d-42d811121213-kube-api-access-5lvrn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-n92qw\" (UID: \"982a22ec-5768-4a3c-843d-42d811121213\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.556765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.556744 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:40.600158 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.600122 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:40.980329 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:40.980296 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5"] Oct 30 22:50:41.024124 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:50:41.024091 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ffecbd_ee8d_4e9f_a7bf_f4348f280830.slice/crio-ab023e037ea9e32a0876f1af47c5b6f247ed82fb7ce0015f73003cbd6faf2b43 WatchSource:0}: Error finding container ab023e037ea9e32a0876f1af47c5b6f247ed82fb7ce0015f73003cbd6faf2b43: Status 404 returned error can't find the container with id ab023e037ea9e32a0876f1af47c5b6f247ed82fb7ce0015f73003cbd6faf2b43 Oct 30 22:50:41.028857 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.028823 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.028968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.028895 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.028968 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.028925 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.035352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.035327 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw"] Oct 30 22:50:41.042374 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:50:41.042353 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982a22ec_5768_4a3c_843d_42d811121213.slice/crio-4a01b960dd4c8e5160a86f2eae86cec99bf0ae19b3dea9b3a901009765ae5a08 WatchSource:0}: Error finding container 4a01b960dd4c8e5160a86f2eae86cec99bf0ae19b3dea9b3a901009765ae5a08: Status 404 returned error can't find the container with id 4a01b960dd4c8e5160a86f2eae86cec99bf0ae19b3dea9b3a901009765ae5a08 Oct 30 22:50:41.044713 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.044685 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.044844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.044743 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.044844 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.044776 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:50:41.147464 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.147419 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" event={"ID":"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830","Type":"ContainerStarted","Data":"a182346d4b6b8889fc6323a4436bac19b55aa9a6ed925c82703cdcb6706c1d53"} Oct 30 22:50:41.147662 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.147518 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" event={"ID":"a1ffecbd-ee8d-4e9f-a7bf-f4348f280830","Type":"ContainerStarted","Data":"ab023e037ea9e32a0876f1af47c5b6f247ed82fb7ce0015f73003cbd6faf2b43"} Oct 30 22:50:41.150408 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.150379 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" event={"ID":"982a22ec-5768-4a3c-843d-42d811121213","Type":"ContainerStarted","Data":"e2e810215ad83961ce7f2e63db7a5ef17e853fd7f183e5c46ebefb7e41bb3839"} Oct 30 22:50:41.150558 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.150430 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" event={"ID":"982a22ec-5768-4a3c-843d-42d811121213","Type":"ContainerStarted","Data":"4a01b960dd4c8e5160a86f2eae86cec99bf0ae19b3dea9b3a901009765ae5a08"} Oct 30 22:50:41.262509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.262406 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podStartSLOduration=1.262388278 podStartE2EDuration="1.262388278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:41.261624751 +0000 UTC m=+2961.856713080" watchObservedRunningTime="2026-10-30 22:50:41.262388278 +0000 UTC m=+2961.857476584" Oct 30 22:50:41.376927 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.376860 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podStartSLOduration=1.37684036 podStartE2EDuration="1.37684036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:41.375469772 +0000 UTC m=+2961.970558079" watchObservedRunningTime="2026-10-30 22:50:41.37684036 +0000 UTC m=+2961.971928669" Oct 30 22:50:41.557155 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.557044 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:41.610822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:41.605926 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:42.561821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.555865 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:42.561821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.556136 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8cf787766-tt8cp" podUID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" containerName="manager" containerID="cri-o://2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b" gracePeriod=30 Oct 30 22:50:42.585317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.571240 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:42.585317 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.571310 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:42.602433 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.602396 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:42.602604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.602458 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:42.922075 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:42.922051 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:43.014470 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.014439 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\") pod \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " Oct 30 22:50:43.014671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.014660 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\") pod \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " Oct 30 22:50:43.014815 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.014784 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\" (UniqueName: \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\") pod \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\" (UID: \"7ff25959-2b8f-49f0-b1b0-db5dd688f410\") " Oct 30 22:50:43.016640 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.016612 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs" pod "7ff25959-2b8f-49f0-b1b0-db5dd688f410" (UID: "7ff25959-2b8f-49f0-b1b0-db5dd688f410"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:43.016776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.016684 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert" pod "7ff25959-2b8f-49f0-b1b0-db5dd688f410" (UID: "7ff25959-2b8f-49f0-b1b0-db5dd688f410"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:43.016776 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.016697 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc" pod "7ff25959-2b8f-49f0-b1b0-db5dd688f410" (UID: "7ff25959-2b8f-49f0-b1b0-db5dd688f410"). InnerVolumeSpecName "kube-api-access-hvgqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:43.116445 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.116406 4305 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-webhook-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:43.116671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.116639 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff25959-2b8f-49f0-b1b0-db5dd688f410-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:43.116671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.116657 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvgqc\" (UniqueName: \"kubernetes.io/projected/7ff25959-2b8f-49f0-b1b0-db5dd688f410-kube-api-access-hvgqc\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:43.203924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.203888 4305 generic.go:356] "Generic (PLEG): container finished" podID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" containerID="2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b" exitCode=0 Oct 30 22:50:43.204099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.204030 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-tt8cp" Oct 30 22:50:43.204099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.204070 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-tt8cp" event={"ID":"7ff25959-2b8f-49f0-b1b0-db5dd688f410","Type":"ContainerDied","Data":"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b"} Oct 30 22:50:43.204191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.204136 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-tt8cp" event={"ID":"7ff25959-2b8f-49f0-b1b0-db5dd688f410","Type":"ContainerDied","Data":"08c8e2bd3284a7900fd7cc1357d1de89db3875166f81ea32276e01e9b24b072e"} Oct 30 22:50:43.204191 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.204168 4305 scope.go:122] "RemoveContainer" containerID="2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b" Oct 30 22:50:43.214204 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.214185 4305 scope.go:122] "RemoveContainer" containerID="2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b" Oct 30 22:50:43.214432 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:50:43.214415 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b\": container with ID starting with 2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b not found: ID does not exist" containerID="2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b" Oct 30 22:50:43.214506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.214445 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b"} err="failed to get container status \"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b\": rpc error: code = NotFound desc = could not find container \"2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b\": container with ID starting with 2e242dd2b1a56476f80b9bb641d61fe67724fde48e06c8c538a3e8dcac70a34b not found: ID does not exist" Oct 30 22:50:43.391932 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.391854 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:43.394872 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.393912 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8cf787766-tt8cp"] Oct 30 22:50:43.517035 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.516950 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" path="/var/lib/kubelet/pods/7ff25959-2b8f-49f0-b1b0-db5dd688f410/volumes" Oct 30 22:50:43.560830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.560230 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:43.560830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.560308 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:43.602197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.602150 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:43.602197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:43.602200 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:44.564823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.559218 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:44.564823 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.559279 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:44.602416 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.602366 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:44.602825 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.602431 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:44.934106 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.934036 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8cf787766-76dxg"] Oct 30 22:50:44.934453 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.934439 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" containerName="manager" Oct 30 22:50:44.934494 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.934454 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" containerName="manager" Oct 30 22:50:44.934529 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.934509 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff25959-2b8f-49f0-b1b0-db5dd688f410" containerName="manager" Oct 30 22:50:44.938748 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:44.938728 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.034771 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.034734 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-webhook-cert\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.034979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.034785 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4fd\" (UniqueName: \"kubernetes.io/projected/15e59070-1878-47e8-9e79-8528184385f9-kube-api-access-2f4fd\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.034979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.034935 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-metrics-certs\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.036630 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.036603 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ct9hv\"" Oct 30 22:50:45.036740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.036636 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:50:45.036740 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.036671 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:50:45.036852 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.036840 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:50:45.044288 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.044262 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-76dxg"] Oct 30 22:50:45.136178 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.136146 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-metrics-certs\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.136384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.136214 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-webhook-cert\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.136384 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.136277 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4fd\" (UniqueName: \"kubernetes.io/projected/15e59070-1878-47e8-9e79-8528184385f9-kube-api-access-2f4fd\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.138819 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.138729 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-webhook-cert\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.138928 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.138821 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e59070-1878-47e8-9e79-8528184385f9-metrics-certs\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.199233 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.199196 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4fd\" (UniqueName: \"kubernetes.io/projected/15e59070-1878-47e8-9e79-8528184385f9-kube-api-access-2f4fd\") pod \"maas-controller-8cf787766-76dxg\" (UID: \"15e59070-1878-47e8-9e79-8528184385f9\") " pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.285822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.284971 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:45.568884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.558280 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:45.568884 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.558340 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:45.600936 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.600888 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:45.601131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.600962 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:45.673821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:45.672932 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8cf787766-76dxg"] Oct 30 22:50:46.120052 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:50:46.120006 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:50:46.222297 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:46.222262 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-76dxg" event={"ID":"15e59070-1878-47e8-9e79-8528184385f9","Type":"ContainerStarted","Data":"abd56fd03cf2bb42f0a9ba4007c341ad6920f6141ce384385491fcc14309f601"} Oct 30 22:50:46.562820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:46.559221 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:46.562820 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:46.559284 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:46.601031 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:46.600988 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:46.601203 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:46.601051 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:47.228312 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.228272 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8cf787766-76dxg" event={"ID":"15e59070-1878-47e8-9e79-8528184385f9","Type":"ContainerStarted","Data":"64e1610eeb1b180cdbc1394d2a44baf21611eb6d7b5ff0eb83fb5cc46e2d9f67"} Oct 30 22:50:47.228853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.228395 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:50:47.345296 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.345231 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8cf787766-76dxg" podStartSLOduration=2.4023553189999998 podStartE2EDuration="3.345210202s" totalImagesPullingTime="942.854883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:47.343537453 +0000 UTC m=+2967.938625762" watchObservedRunningTime="2026-10-30 22:50:47.345210202 +0000 UTC m=+2967.940298510" Oct 30 22:50:47.557753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.557668 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:47.557753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.557720 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:47.600866 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.600830 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:47.601032 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:47.600887 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:48.558283 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:48.558239 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cg4z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 22:50:48.558668 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:48.558311 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" podUID="a1ffecbd-ee8d-4e9f-a7bf-f4348f280830" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 22:50:48.601451 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:48.601416 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-n92qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:50:48.601594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:48.601481 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" podUID="982a22ec-5768-4a3c-843d-42d811121213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:50:49.561882 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.561848 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:49.562361 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.562258 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:49.563024 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.563002 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cg4z5" Oct 30 22:50:49.604779 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.604745 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:49.605215 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.605196 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:49.605877 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:49.605851 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-n92qw" Oct 30 22:50:58.235633 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:50:58.235600 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8cf787766-76dxg" Oct 30 22:51:16.108299 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:51:16.108265 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:51:19.918422 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:19.918392 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:51:19.925320 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:19.925299 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:51:40.720065 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.719975 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw"] Oct 30 22:51:40.723719 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.723693 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904221 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904191 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904240 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904272 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904388 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904362 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904398 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904506 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904518 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904570 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904554 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64cv\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-kube-api-access-h64cv\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.904681 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.904569 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:40.925024 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:40.924996 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw"] Oct 30 22:51:41.005376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005306 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005339 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005376 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005370 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005438 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005647 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005605 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64cv\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-kube-api-access-h64cv\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005664 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005767 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005746 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005767 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005824 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.005963 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.005872 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.006079 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.006041 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.006175 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.006152 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.006447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.006425 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.007683 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.007662 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.007956 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.007936 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.064042 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.064014 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.064131 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.064107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64cv\" (UniqueName: \"kubernetes.io/projected/0f68ce5c-3d8d-418a-8ecf-563c339c1e0c-kube-api-access-h64cv\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4kprw\" (UID: \"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.339901 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.339820 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:41.764730 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.764692 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw"] Oct 30 22:51:41.770779 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:51:41.770752 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f68ce5c_3d8d_418a_8ecf_563c339c1e0c.slice/crio-0a564bf1c8c9b1a0bfe646a073d3be6ab2e17d6095ce647fadad53f85461dc6a WatchSource:0}: Error finding container 0a564bf1c8c9b1a0bfe646a073d3be6ab2e17d6095ce647fadad53f85461dc6a: Status 404 returned error can't find the container with id 0a564bf1c8c9b1a0bfe646a073d3be6ab2e17d6095ce647fadad53f85461dc6a Oct 30 22:51:41.772912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.772880 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:41.772987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.772944 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:41.772987 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:41.772974 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:42.483098 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:42.483062 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" event={"ID":"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c","Type":"ContainerStarted","Data":"e0ed67435a6547fece7475853ffa3cc00711d3ee2b12678aac7c1856989c9dc1"} Oct 30 22:51:42.483312 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:42.483118 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" event={"ID":"0f68ce5c-3d8d-418a-8ecf-563c339c1e0c","Type":"ContainerStarted","Data":"0a564bf1c8c9b1a0bfe646a073d3be6ab2e17d6095ce647fadad53f85461dc6a"} Oct 30 22:51:42.658822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:42.651037 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podStartSLOduration=2.651018287 podStartE2EDuration="2.651018287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.64562408 +0000 UTC m=+3023.240712409" watchObservedRunningTime="2026-10-30 22:51:42.651018287 +0000 UTC m=+3023.246106596" Oct 30 22:51:43.340906 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:43.340869 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:43.342575 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:43.342548 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:43.342709 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:43.342595 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:44.341197 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:44.341159 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:44.341631 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:44.341216 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:45.340659 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:45.340622 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:45.340948 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:45.340679 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:46.110020 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:51:46.109907 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:51:46.341066 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:46.341024 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:46.341275 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:46.341085 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:47.341064 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:47.341024 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:47.341479 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:47.341082 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:48.340813 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:48.340748 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4kprw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:51:48.340999 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:48.340850 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" podUID="0f68ce5c-3d8d-418a-8ecf-563c339c1e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:51:49.973069 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:49.973034 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:49.973580 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:49.973386 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:51:49.974258 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:51:49.974236 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4kprw" Oct 30 22:52:16.108271 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:52:16.108235 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:52:46.109543 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:52:46.109505 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:53:16.108554 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:53:16.108513 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:53:46.108862 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:53:46.108820 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:54:16.109442 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:54:16.109408 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:54:46.108427 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:54:46.108337 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:55:16.108361 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:55:16.108324 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:55:46.108743 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:55:46.108701 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:56:16.109122 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:56:16.109085 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:56:19.947038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:56:19.947009 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:56:19.953624 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:56:19.953603 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 22:56:46.109042 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:56:46.109003 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:57:16.109487 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:57:16.109449 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:57:45.993077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:45.992996 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:57:45.996746 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:45.996724 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.048957 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.048920 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dcc58b-openshift-default-dockercfg-99tcm\"" Oct 30 22:57:46.080487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080453 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080487 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080491 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ck8d\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080521 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080645 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080696 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080684 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080847 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080818 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080888 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080869 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080924 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080911 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.080983 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.080970 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.101514 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.101484 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:57:46.113698 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:57:46.113662 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182563 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182626 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182669 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182702 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182785 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182830 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ck8d\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182878 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182909 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.182944 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.183438 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.183848 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.183708 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.185245 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.185219 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.185410 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.185340 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.185743 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.185710 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.188272 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.188245 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.190499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.190470 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.242996 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.242970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.243210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.243166 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ck8d\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\") pod \"e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.309625 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.309591 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:46.733099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.733066 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:57:46.739829 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:57:46.739779 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d4b313_cb38_4770_abfb_563648a3be22.slice/crio-cff3aad9dd3a92cce59a8b503d5446cbcf40009140d2b90c778e16209eddcaa0 WatchSource:0}: Error finding container cff3aad9dd3a92cce59a8b503d5446cbcf40009140d2b90c778e16209eddcaa0: Status 404 returned error can't find the container with id cff3aad9dd3a92cce59a8b503d5446cbcf40009140d2b90c778e16209eddcaa0 Oct 30 22:57:46.742103 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.742072 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:57:46.742179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.742139 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:57:46.742179 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:46.742166 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:57:47.326671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:47.326624 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerStarted","Data":"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b"} Oct 30 22:57:47.327163 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:47.326731 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerStarted","Data":"cff3aad9dd3a92cce59a8b503d5446cbcf40009140d2b90c778e16209eddcaa0"} Oct 30 22:57:47.441471 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:47.441421 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podStartSLOduration=2.441407131 podStartE2EDuration="2.441407131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:47.440083129 +0000 UTC m=+3388.035171427" watchObservedRunningTime="2026-10-30 22:57:47.441407131 +0000 UTC m=+3388.036495436" Oct 30 22:57:48.310830 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:48.310774 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:48.315855 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:48.315827 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:48.331712 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:48.331683 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:48.332533 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:48.332516 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:57:54.025095 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.025058 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:57:54.028532 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.028503 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.078008 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.077980 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dcc58b\"" Oct 30 22:57:54.123750 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.123722 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-471e3a52-dockercfg-zmshq\"" Oct 30 22:57:54.125705 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.125683 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:57:54.158457 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.158421 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.158607 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.158565 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xs4\" (UniqueName: \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.259450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.259415 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.259671 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.259493 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xs4\" (UniqueName: \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.260277 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.260249 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.317266 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.317195 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xs4\" (UniqueName: \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\") pod \"payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.341978 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.341948 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:54.711916 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:54.711881 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:57:54.719220 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:57:54.719191 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde704cf6_aa3d_4301_9aeb_b27908127ae6.slice/crio-3437a49c7d721c13969b6fcab6eb32381d025e3bbd6963a03b800149073621dc WatchSource:0}: Error finding container 3437a49c7d721c13969b6fcab6eb32381d025e3bbd6963a03b800149073621dc: Status 404 returned error can't find the container with id 3437a49c7d721c13969b6fcab6eb32381d025e3bbd6963a03b800149073621dc Oct 30 22:57:55.363176 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:55.363141 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" event={"ID":"de704cf6-aa3d-4301-9aeb-b27908127ae6","Type":"ContainerStarted","Data":"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1"} Oct 30 22:57:55.363574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:55.363191 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" event={"ID":"de704cf6-aa3d-4301-9aeb-b27908127ae6","Type":"ContainerStarted","Data":"3437a49c7d721c13969b6fcab6eb32381d025e3bbd6963a03b800149073621dc"} Oct 30 22:57:55.363574 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:55.363282 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:57:55.472604 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:55.472547 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" podStartSLOduration=2.472530754 podStartE2EDuration="2.472530754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:55.472163259 +0000 UTC m=+3396.067251566" watchObservedRunningTime="2026-10-30 22:57:55.472530754 +0000 UTC m=+3396.067619060" Oct 30 22:57:56.096144 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.096111 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:57:56.099902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.099880 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.148895 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.148866 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dcc58b-serving-cert\"" Oct 30 22:57:56.180015 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.179963 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.180146 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.180020 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.180198 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.180158 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdjj\" (UniqueName: \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.180291 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.180272 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.196649 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.196616 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:57:56.281395 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.281351 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdjj\" (UniqueName: \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.281588 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.281527 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.281675 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.281658 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.281735 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.281695 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.281786 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:57:56.281771 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-serving-cert: secret "maas-api-e2e-worker-single-main-dcc58b-serving-cert" not found Oct 30 22:57:56.281890 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:57:56.281857 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls podName:32243eb3-a483-440c-b5cb-df9c4b3adbc5 nodeName:}" failed. No retries permitted until 2026-10-30 22:57:56.781838308 +0000 UTC m=+3397.376926597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls") pod "maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" (UID: "32243eb3-a483-440c-b5cb-df9c4b3adbc5") : secret "maas-api-e2e-worker-single-main-dcc58b-serving-cert" not found Oct 30 22:57:56.282352 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.282327 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.284007 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.283987 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.341222 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.341191 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdjj\" (UniqueName: \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.787027 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.786984 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:56.789461 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:56.789439 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:57.013001 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:57.012968 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:57.386192 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:57.386111 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:57:57.392200 ip-10-0-134-125 kubenswrapper[4305]: W1030 22:57:57.392173 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32243eb3_a483_440c_b5cb_df9c4b3adbc5.slice/crio-7fee39a572ba79149c108321ae69e4d62f7ea4682289075746c48cfcde922758 WatchSource:0}: Error finding container 7fee39a572ba79149c108321ae69e4d62f7ea4682289075746c48cfcde922758: Status 404 returned error can't find the container with id 7fee39a572ba79149c108321ae69e4d62f7ea4682289075746c48cfcde922758 Oct 30 22:57:57.393599 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:57.393582 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:57:58.377998 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:58.377962 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" event={"ID":"32243eb3-a483-440c-b5cb-df9c4b3adbc5","Type":"ContainerStarted","Data":"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99"} Oct 30 22:57:58.378450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:58.378022 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" event={"ID":"32243eb3-a483-440c-b5cb-df9c4b3adbc5","Type":"ContainerStarted","Data":"7fee39a572ba79149c108321ae69e4d62f7ea4682289075746c48cfcde922758"} Oct 30 22:57:58.378450 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:58.378052 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:57:58.496126 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:57:58.496059 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" podStartSLOduration=1.610334175 podStartE2EDuration="2.49603892s" totalImagesPullingTime="885.704745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:58.495584248 +0000 UTC m=+3399.090672554" watchObservedRunningTime="2026-10-30 22:57:58.49603892 +0000 UTC m=+3399.091127227" Oct 30 22:58:04.389367 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:04.389334 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:58:06.371720 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:06.371685 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:58:13.448697 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.448663 4305 generic.go:356] "Generic (PLEG): container finished" podID="61d4b313-cb38-4770-abfb-563648a3be22" containerID="ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b" exitCode=0 Oct 30 22:58:13.449211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.448739 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerDied","Data":"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b"} Oct 30 22:58:13.449211 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.449170 4305 scope.go:122] "RemoveContainer" containerID="ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b" Oct 30 22:58:13.450448 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.450424 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:58:13.450510 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.450497 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:58:13.450549 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:13.450533 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:58:14.478322 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:14.478286 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerStarted","Data":"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953"} Oct 30 22:58:15.310650 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:15.310610 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:58:15.311907 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:15.311881 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:58:15.311981 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:15.311933 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:58:16.109362 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:58:16.109328 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:58:16.310821 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:16.310758 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:58:16.311002 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:16.310847 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:58:17.310252 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:17.310213 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:58:17.310734 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:17.310282 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:58:18.309967 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:18.309932 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:58:18.310154 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:18.309985 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:58:18.332162 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:18.332128 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:58:19.310879 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:19.310835 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 22:58:19.311068 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:19.310898 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 22:58:20.314620 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:20.314585 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:58:20.315594 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:58:20.315571 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:58:46.110243 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:58:46.110202 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:59:16.109146 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:16.109109 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:59:27.022973 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:27.022935 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:59:27.900499 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:27.900465 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:59:27.900704 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:27.900681 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" podUID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" containerName="maas-api" containerID="cri-o://43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99" gracePeriod=30 Oct 30 22:59:28.363289 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.363266 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:59:28.503368 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.503337 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") pod \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " Oct 30 22:59:28.503569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.503383 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\" (UniqueName: \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\") pod \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " Oct 30 22:59:28.503569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.503497 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\") pod \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " Oct 30 22:59:28.503569 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.503532 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\") pod \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\" (UID: \"32243eb3-a483-440c-b5cb-df9c4b3adbc5\") " Oct 30 22:59:28.504152 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.504124 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca" pod "32243eb3-a483-440c-b5cb-df9c4b3adbc5" (UID: "32243eb3-a483-440c-b5cb-df9c4b3adbc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:28.505648 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.505617 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj" pod "32243eb3-a483-440c-b5cb-df9c4b3adbc5" (UID: "32243eb3-a483-440c-b5cb-df9c4b3adbc5"). InnerVolumeSpecName "kube-api-access-dqdjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:28.505986 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.505964 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs" pod "32243eb3-a483-440c-b5cb-df9c4b3adbc5" (UID: "32243eb3-a483-440c-b5cb-df9c4b3adbc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:59:28.506085 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.505986 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls" pod "32243eb3-a483-440c-b5cb-df9c4b3adbc5" (UID: "32243eb3-a483-440c-b5cb-df9c4b3adbc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:59:28.605220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.605189 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:28.605220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.605215 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdjj\" (UniqueName: \"kubernetes.io/projected/32243eb3-a483-440c-b5cb-df9c4b3adbc5-kube-api-access-dqdjj\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:28.605220 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.605225 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32243eb3-a483-440c-b5cb-df9c4b3adbc5-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:28.605447 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.605234 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32243eb3-a483-440c-b5cb-df9c4b3adbc5-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:28.821912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.821821 4305 generic.go:356] "Generic (PLEG): container finished" podID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" containerID="43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99" exitCode=0 Oct 30 22:59:28.821912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.821857 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" Oct 30 22:59:28.821912 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.821894 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" event={"ID":"32243eb3-a483-440c-b5cb-df9c4b3adbc5","Type":"ContainerDied","Data":"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99"} Oct 30 22:59:28.822182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.821950 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm" event={"ID":"32243eb3-a483-440c-b5cb-df9c4b3adbc5","Type":"ContainerDied","Data":"7fee39a572ba79149c108321ae69e4d62f7ea4682289075746c48cfcde922758"} Oct 30 22:59:28.822182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.821974 4305 scope.go:122] "RemoveContainer" containerID="43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99" Oct 30 22:59:28.834674 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.834649 4305 scope.go:122] "RemoveContainer" containerID="43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99" Oct 30 22:59:28.835060 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:28.835029 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99\": container with ID starting with 43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99 not found: ID does not exist" containerID="43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99" Oct 30 22:59:28.835183 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.835062 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99"} err="failed to get container status \"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99\": rpc error: code = NotFound desc = could not find container \"43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99\": container with ID starting with 43bb7fda55298c736bbc62103e6cf4f37c26cef3bfabb95862761338d4e4dc99 not found: ID does not exist" Oct 30 22:59:28.977853 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.977816 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:59:28.981573 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:28.981545 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dcc58b-86ccccc689-r9cbm"] Oct 30 22:59:29.517380 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:29.517351 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" path="/var/lib/kubelet/pods/32243eb3-a483-440c-b5cb-df9c4b3adbc5/volumes" Oct 30 22:59:32.023652 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:32.023600 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" containerID="cri-o://284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1" gracePeriod=25 Oct 30 22:59:36.369358 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:36.369305 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:59:39.124920 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:39.124887 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:59:39.125444 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:39.125105 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" containerID="cri-o://7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953" gracePeriod=30 Oct 30 22:59:44.380507 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.380478 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:59:44.459112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459079 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459112 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459115 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459157 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459351 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459292 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459352 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459396 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459466 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459436 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459472 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:44.459621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459497 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459518 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:44.459621 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459569 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\") pod \"61d4b313-cb38-4770-abfb-563648a3be22\" (UID: \"61d4b313-cb38-4770-abfb-563648a3be22\") " Oct 30 22:59:44.459842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459640 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:44.459842 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459788 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:44.459979 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459960 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.460038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.459993 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4b313-cb38-4770-abfb-563648a3be22-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.460038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.460006 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.460038 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.460016 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.460210 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.460144 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:44.461491 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.461465 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:59:44.461595 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.461570 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:44.461832 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.461788 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:44.461902 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.461889 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d" pod "61d4b313-cb38-4770-abfb-563648a3be22" (UID: "61d4b313-cb38-4770-abfb-563648a3be22"). InnerVolumeSpecName "kube-api-access-6ck8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:44.560509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.560472 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ck8d\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-kube-api-access-6ck8d\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.560509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.560502 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4b313-cb38-4770-abfb-563648a3be22-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.560509 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.560512 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.560753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.560520 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4b313-cb38-4770-abfb-563648a3be22-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.560753 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.560531 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4b313-cb38-4770-abfb-563648a3be22-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:44.897099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.897019 4305 generic.go:356] "Generic (PLEG): container finished" podID="61d4b313-cb38-4770-abfb-563648a3be22" containerID="7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953" exitCode=0 Oct 30 22:59:44.897099 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.897068 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" Oct 30 22:59:44.897290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.897104 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerDied","Data":"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953"} Oct 30 22:59:44.897290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.897172 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k" event={"ID":"61d4b313-cb38-4770-abfb-563648a3be22","Type":"ContainerDied","Data":"cff3aad9dd3a92cce59a8b503d5446cbcf40009140d2b90c778e16209eddcaa0"} Oct 30 22:59:44.897290 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.897200 4305 scope.go:122] "RemoveContainer" containerID="7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953" Oct 30 22:59:44.907933 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.907911 4305 scope.go:122] "RemoveContainer" containerID="ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b" Oct 30 22:59:44.916182 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.916159 4305 scope.go:122] "RemoveContainer" containerID="7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953" Oct 30 22:59:44.916428 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:44.916408 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953\": container with ID starting with 7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953 not found: ID does not exist" containerID="7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953" Oct 30 22:59:44.916493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.916438 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953"} err="failed to get container status \"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953\": rpc error: code = NotFound desc = could not find container \"7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953\": container with ID starting with 7853dce15a9e17d1c99d800aac2923e1229c8d6881073b6b69eac8ea74c71953 not found: ID does not exist" Oct 30 22:59:44.916493 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.916457 4305 scope.go:122] "RemoveContainer" containerID="ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b" Oct 30 22:59:44.916757 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:44.916736 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b\": container with ID starting with ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b not found: ID does not exist" containerID="ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b" Oct 30 22:59:44.916831 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:44.916769 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b"} err="failed to get container status \"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b\": rpc error: code = NotFound desc = could not find container \"ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b\": container with ID starting with ba581d68e13eeafa3cc89ca4b3a285acf44ff58acf7a70ab99f7247f68eb566b not found: ID does not exist" Oct 30 22:59:45.053765 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:45.053729 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:59:45.099839 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:45.099791 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dcc58b-openshift-default-75b6945cc-tk98k"] Oct 30 22:59:45.517311 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:45.517277 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d4b313-cb38-4770-abfb-563648a3be22" path="/var/lib/kubelet/pods/61d4b313-cb38-4770-abfb-563648a3be22/volumes" Oct 30 22:59:46.109033 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:46.108989 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 22:59:46.369426 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:46.369332 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:59:52.273725 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.273700 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:59:52.332698 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.332633 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\") pod \"de704cf6-aa3d-4301-9aeb-b27908127ae6\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " Oct 30 22:59:52.332843 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.332793 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\" (UniqueName: \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\") pod \"de704cf6-aa3d-4301-9aeb-b27908127ae6\" (UID: \"de704cf6-aa3d-4301-9aeb-b27908127ae6\") " Oct 30 22:59:52.333053 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.333026 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume" pod "de704cf6-aa3d-4301-9aeb-b27908127ae6" (UID: "de704cf6-aa3d-4301-9aeb-b27908127ae6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:59:52.334824 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.334783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4" pod "de704cf6-aa3d-4301-9aeb-b27908127ae6" (UID: "de704cf6-aa3d-4301-9aeb-b27908127ae6"). InnerVolumeSpecName "kube-api-access-t7xs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:52.434227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.434198 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xs4\" (UniqueName: \"kubernetes.io/projected/de704cf6-aa3d-4301-9aeb-b27908127ae6-kube-api-access-t7xs4\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:52.434227 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.434224 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de704cf6-aa3d-4301-9aeb-b27908127ae6-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:52.936303 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.936268 4305 generic.go:356] "Generic (PLEG): container finished" podID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerID="284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1" exitCode=0 Oct 30 22:59:52.936478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.936319 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" Oct 30 22:59:52.936478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.936351 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" event={"ID":"de704cf6-aa3d-4301-9aeb-b27908127ae6","Type":"ContainerDied","Data":"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1"} Oct 30 22:59:52.936478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.936411 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr" event={"ID":"de704cf6-aa3d-4301-9aeb-b27908127ae6","Type":"ContainerDied","Data":"3437a49c7d721c13969b6fcab6eb32381d025e3bbd6963a03b800149073621dc"} Oct 30 22:59:52.936478 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.936432 4305 scope.go:122] "RemoveContainer" containerID="284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1" Oct 30 22:59:52.945515 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.945494 4305 scope.go:122] "RemoveContainer" containerID="284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1" Oct 30 22:59:52.945774 ip-10-0-134-125 kubenswrapper[4305]: E1030 22:59:52.945755 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1\": container with ID starting with 284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1 not found: ID does not exist" containerID="284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1" Oct 30 22:59:52.945854 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:52.945781 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1"} err="failed to get container status \"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1\": rpc error: code = NotFound desc = could not find container \"284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1\": container with ID starting with 284ab96c634535f4e4f95795ec56b1b9703fc943c9515f6de8959c11b7add2c1 not found: ID does not exist" Oct 30 22:59:53.092150 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:53.092118 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:59:53.138077 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:53.138042 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dcc58b-5487f988dsqmrr"] Oct 30 22:59:53.516822 ip-10-0-134-125 kubenswrapper[4305]: I1030 22:59:53.516759 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" path="/var/lib/kubelet/pods/de704cf6-aa3d-4301-9aeb-b27908127ae6/volumes" Oct 30 23:00:16.108554 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:00:16.108514 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:00:46.108547 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:00:46.108512 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:01:16.108970 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:01:16.108925 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:01:19.975568 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:01:19.975536 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:01:19.982771 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:01:19.982749 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:01:46.109340 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:01:46.109302 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:02:16.108962 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:02:16.108925 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:02:26.739377 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739344 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739706 4305 cpu_manager.go:414] "containerMap: removing container" podUID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" containerName="maas-api" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739716 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" containerName="maas-api" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739728 4305 cpu_manager.go:414] "containerMap: removing container" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739733 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739749 4305 cpu_manager.go:414] "containerMap: removing container" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739754 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739848 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="de704cf6-aa3d-4301-9aeb-b27908127ae6" containerName="payload-processing" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739862 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739869 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="32243eb3-a483-440c-b5cb-df9c4b3adbc5" containerName="maas-api" Oct 30 23:02:26.739882 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.739874 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:26.743138 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.743117 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.768957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.768874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.768957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.768942 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769132 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769035 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9mg\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769132 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769077 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769261 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769196 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769261 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769248 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769348 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769290 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769398 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769356 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.769398 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.769380 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.794101 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.794079 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2e092e-openshift-default-dockercfg-wjh5n\"" Oct 30 23:02:26.850103 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.850074 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:02:26.870684 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.870648 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.870840 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.870710 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.870840 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.870767 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871067 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871040 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871176 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871111 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871176 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871169 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871298 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871210 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9mg\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871298 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871255 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871298 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871282 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871457 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871314 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871457 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871390 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871892 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871892 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871494 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.871892 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.871638 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.873751 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.873728 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.873914 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.873896 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.940433 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.940397 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:26.940633 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:26.940575 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9mg\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\") pod \"e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:27.062180 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.062068 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:27.492759 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.492724 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:02:27.499115 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:02:27.499087 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3194315_82b4_414a_9f65_73a1afa9efb6.slice/crio-1bd30dc6125f7f81be458cbb04fb08879116007d76eb427dfee6f4b6205508d5 WatchSource:0}: Error finding container 1bd30dc6125f7f81be458cbb04fb08879116007d76eb427dfee6f4b6205508d5: Status 404 returned error can't find the container with id 1bd30dc6125f7f81be458cbb04fb08879116007d76eb427dfee6f4b6205508d5 Oct 30 23:02:27.501146 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.501112 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:27.501218 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.501182 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:27.501256 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.501216 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:27.642652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.642611 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerStarted","Data":"b56b55c5cbc06bdd26ad396845ebeb34348c788c16c8c2dbe8e060ff141b63c8"} Oct 30 23:02:27.642835 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.642681 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerStarted","Data":"1bd30dc6125f7f81be458cbb04fb08879116007d76eb427dfee6f4b6205508d5"} Oct 30 23:02:27.758377 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:27.758276 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podStartSLOduration=1.758262719 podStartE2EDuration="1.758262719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:27.75596442 +0000 UTC m=+3668.351052716" watchObservedRunningTime="2026-10-30 23:02:27.758262719 +0000 UTC m=+3668.353351025" Oct 30 23:02:28.063683 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:28.063603 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:28.068424 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:28.068398 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:28.648042 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:28.648006 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:28.648866 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:28.648842 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:34.431521 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.431482 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:02:34.432022 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.431935 4305 cpu_manager.go:414] "containerMap: removing container" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:34.432022 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.431950 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d4b313-cb38-4770-abfb-563648a3be22" containerName="istio-proxy" Oct 30 23:02:34.435386 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.435361 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.530410 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.530378 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2e092e\"" Oct 30 23:02:34.530410 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.530402 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b38a71aa-dockercfg-v9wc2\"" Oct 30 23:02:34.533562 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.533528 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:02:34.558250 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.558223 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.558418 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.558304 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jh8k\" (UniqueName: \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.659848 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.659815 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.660005 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.659878 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh8k\" (UniqueName: \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.660484 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.660463 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.717677 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.717640 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh8k\" (UniqueName: \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\") pod \"payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:34.748235 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:34.748207 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:35.121634 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:35.121561 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:02:35.128137 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:02:35.128105 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419b82e5_fc69_462f_b436_c004b984ac4f.slice/crio-c555f96000a3697a27b05350246e93afdd8fcfe135bfb9729874a33e4e330406 WatchSource:0}: Error finding container c555f96000a3697a27b05350246e93afdd8fcfe135bfb9729874a33e4e330406: Status 404 returned error can't find the container with id c555f96000a3697a27b05350246e93afdd8fcfe135bfb9729874a33e4e330406 Oct 30 23:02:35.679952 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:35.679917 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" event={"ID":"419b82e5-fc69-462f-b436-c004b984ac4f","Type":"ContainerStarted","Data":"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22"} Oct 30 23:02:35.680345 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:35.679970 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" event={"ID":"419b82e5-fc69-462f-b436-c004b984ac4f","Type":"ContainerStarted","Data":"c555f96000a3697a27b05350246e93afdd8fcfe135bfb9729874a33e4e330406"} Oct 30 23:02:35.680345 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:35.680001 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:35.818524 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:35.818472 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" podStartSLOduration=1.81845886 podStartE2EDuration="1.81845886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:35.817007216 +0000 UTC m=+3676.412095522" watchObservedRunningTime="2026-10-30 23:02:35.81845886 +0000 UTC m=+3676.413547169" Oct 30 23:02:36.503117 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.503081 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:02:36.506951 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.506928 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.556479 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.556455 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2e092e-serving-cert\"" Oct 30 23:02:36.580211 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.580182 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.580351 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.580220 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbxk\" (UniqueName: \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.580351 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.580256 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.580540 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.580434 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.603992 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.603964 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:02:36.681627 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.681594 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.682090 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.681725 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.682090 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.681759 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbxk\" (UniqueName: \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.682090 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.681793 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.682090 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:02:36.681880 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-serving-cert: secret "maas-api-e2e-worker-single-main-2e092e-serving-cert" not found Oct 30 23:02:36.682090 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:02:36.681959 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls podName:69443d43-fdca-4e17-980d-63e07633d75b nodeName:}" failed. No retries permitted until 2026-10-30 23:02:37.181935277 +0000 UTC m=+3677.777023575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls") pod "maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" (UID: "69443d43-fdca-4e17-980d-63e07633d75b") : secret "maas-api-e2e-worker-single-main-2e092e-serving-cert" not found Oct 30 23:02:36.682355 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.682327 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.684287 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.684266 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:36.751769 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:36.751736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbxk\" (UniqueName: \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:37.187516 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:37.187486 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:37.189993 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:37.189966 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:37.419582 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:37.419553 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:37.792488 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:37.792415 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:02:37.798766 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:02:37.798733 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69443d43_fdca_4e17_980d_63e07633d75b.slice/crio-1b9d03fd3d01cccb0989180d6b991e872060e9f20eebb4a04f8fa7fd8eee25c0 WatchSource:0}: Error finding container 1b9d03fd3d01cccb0989180d6b991e872060e9f20eebb4a04f8fa7fd8eee25c0: Status 404 returned error can't find the container with id 1b9d03fd3d01cccb0989180d6b991e872060e9f20eebb4a04f8fa7fd8eee25c0 Oct 30 23:02:38.695161 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:38.695129 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" event={"ID":"69443d43-fdca-4e17-980d-63e07633d75b","Type":"ContainerStarted","Data":"1b9d03fd3d01cccb0989180d6b991e872060e9f20eebb4a04f8fa7fd8eee25c0"} Oct 30 23:02:39.701409 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:39.701366 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" event={"ID":"69443d43-fdca-4e17-980d-63e07633d75b","Type":"ContainerStarted","Data":"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba"} Oct 30 23:02:39.701816 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:39.701516 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:39.814722 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:39.814663 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" podStartSLOduration=2.897137628 podStartE2EDuration="3.814650984s" totalImagesPullingTime="917.513356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:39.811984907 +0000 UTC m=+3680.407073208" watchObservedRunningTime="2026-10-30 23:02:39.814650984 +0000 UTC m=+3680.409739290" Oct 30 23:02:45.712475 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:45.712440 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:02:46.124412 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:02:46.124323 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:02:46.688520 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:46.688489 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:02:51.759544 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.759471 4305 generic.go:356] "Generic (PLEG): container finished" podID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerID="b56b55c5cbc06bdd26ad396845ebeb34348c788c16c8c2dbe8e060ff141b63c8" exitCode=0 Oct 30 23:02:51.759998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.759545 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerDied","Data":"b56b55c5cbc06bdd26ad396845ebeb34348c788c16c8c2dbe8e060ff141b63c8"} Oct 30 23:02:51.759998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.759942 4305 scope.go:122] "RemoveContainer" containerID="b56b55c5cbc06bdd26ad396845ebeb34348c788c16c8c2dbe8e060ff141b63c8" Oct 30 23:02:51.761171 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.761146 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:51.761257 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.761211 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:51.761257 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:51.761244 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:02:52.774102 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:52.774068 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerStarted","Data":"17012ecc3ded6083b78f3c37219abe119269f1677b5daddd5563f7c142bb524b"} Oct 30 23:02:53.065305 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:53.065213 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:53.065671 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:53.065645 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 23:02:53.065861 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:53.065835 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 23:02:54.063219 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:54.063183 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 23:02:54.063818 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:54.063758 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 23:02:55.063157 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:55.063121 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 23:02:55.063471 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:55.063438 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 23:02:56.071219 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:56.071183 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 23:02:56.071866 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:56.071789 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 23:02:57.063822 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:57.063191 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 23:02:57.063822 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:57.063257 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 23:02:58.067348 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:58.067316 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:58.067751 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:58.067671 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:02:58.068449 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:02:58.068430 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:03:16.108188 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:03:16.108147 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:03:46.109408 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:03:46.109324 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:04:09.554809 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:09.554772 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:04:14.555951 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:14.555896 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" containerID="cri-o://558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22" gracePeriod=25 Oct 30 23:04:16.109958 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:04:16.109921 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:04:16.686478 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:16.686437 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:04:17.196903 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:17.196871 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:04:17.197275 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:17.197065 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" podUID="69443d43-fdca-4e17-980d-63e07633d75b" containerName="maas-api" containerID="cri-o://e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba" gracePeriod=30 Oct 30 23:04:18.443393 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.443366 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:04:18.485746 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.485719 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") pod \"69443d43-fdca-4e17-980d-63e07633d75b\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " Oct 30 23:04:18.485891 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.485758 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\") pod \"69443d43-fdca-4e17-980d-63e07633d75b\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " Oct 30 23:04:18.485891 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.485791 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\") pod \"69443d43-fdca-4e17-980d-63e07633d75b\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " Oct 30 23:04:18.485891 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.485870 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\" (UniqueName: \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\") pod \"69443d43-fdca-4e17-980d-63e07633d75b\" (UID: \"69443d43-fdca-4e17-980d-63e07633d75b\") " Oct 30 23:04:18.486147 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.486115 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca" pod "69443d43-fdca-4e17-980d-63e07633d75b" (UID: "69443d43-fdca-4e17-980d-63e07633d75b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:04:18.487865 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.487838 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk" pod "69443d43-fdca-4e17-980d-63e07633d75b" (UID: "69443d43-fdca-4e17-980d-63e07633d75b"). InnerVolumeSpecName "kube-api-access-hwbxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:04:18.487959 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.487866 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs" pod "69443d43-fdca-4e17-980d-63e07633d75b" (UID: "69443d43-fdca-4e17-980d-63e07633d75b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:04:18.488007 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.487983 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls" pod "69443d43-fdca-4e17-980d-63e07633d75b" (UID: "69443d43-fdca-4e17-980d-63e07633d75b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:04:18.587391 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.587314 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.587391 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.587351 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69443d43-fdca-4e17-980d-63e07633d75b-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.587391 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.587367 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69443d43-fdca-4e17-980d-63e07633d75b-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.587391 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:18.587381 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbxk\" (UniqueName: \"kubernetes.io/projected/69443d43-fdca-4e17-980d-63e07633d75b-kube-api-access-hwbxk\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:19.170908 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.170866 4305 generic.go:356] "Generic (PLEG): container finished" podID="69443d43-fdca-4e17-980d-63e07633d75b" containerID="e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba" exitCode=0 Oct 30 23:04:19.171077 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.170918 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" event={"ID":"69443d43-fdca-4e17-980d-63e07633d75b","Type":"ContainerDied","Data":"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba"} Oct 30 23:04:19.171077 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.170992 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" event={"ID":"69443d43-fdca-4e17-980d-63e07633d75b","Type":"ContainerDied","Data":"1b9d03fd3d01cccb0989180d6b991e872060e9f20eebb4a04f8fa7fd8eee25c0"} Oct 30 23:04:19.171077 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.170935 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d" Oct 30 23:04:19.171077 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.171024 4305 scope.go:122] "RemoveContainer" containerID="e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba" Oct 30 23:04:19.180939 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.180922 4305 scope.go:122] "RemoveContainer" containerID="e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba" Oct 30 23:04:19.181203 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:04:19.181179 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba\": container with ID starting with e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba not found: ID does not exist" containerID="e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba" Oct 30 23:04:19.181304 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.181213 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba"} err="failed to get container status \"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba\": rpc error: code = NotFound desc = could not find container \"e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba\": container with ID starting with e6b237bd69b61e335698733dfc8c7833d45afcacf1e80a829d8d19134ff051ba not found: ID does not exist" Oct 30 23:04:19.326331 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.326263 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:04:19.328273 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.328247 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2e092e-57f8b6b67c-r287d"] Oct 30 23:04:19.518238 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:19.518207 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69443d43-fdca-4e17-980d-63e07633d75b" path="/var/lib/kubelet/pods/69443d43-fdca-4e17-980d-63e07633d75b/volumes" Oct 30 23:04:26.068181 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:26.068140 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:04:26.068679 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:26.068376 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" containerID="cri-o://17012ecc3ded6083b78f3c37219abe119269f1677b5daddd5563f7c142bb524b" gracePeriod=30 Oct 30 23:04:26.686136 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:26.686096 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 23:04:28.069625 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:28.069585 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:04:28.070114 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:28.069661 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:04:31.228009 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.227978 4305 generic.go:356] "Generic (PLEG): container finished" podID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerID="17012ecc3ded6083b78f3c37219abe119269f1677b5daddd5563f7c142bb524b" exitCode=0 Oct 30 23:04:31.228436 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.228048 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerDied","Data":"17012ecc3ded6083b78f3c37219abe119269f1677b5daddd5563f7c142bb524b"} Oct 30 23:04:31.228436 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.228129 4305 scope.go:122] "RemoveContainer" containerID="b56b55c5cbc06bdd26ad396845ebeb34348c788c16c8c2dbe8e060ff141b63c8" Oct 30 23:04:31.321549 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.321523 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:04:31.415622 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415548 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415622 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415601 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415636 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415687 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415720 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415749 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.415842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415821 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.416039 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415840 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:31.416039 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415883 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.416039 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415943 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:31.416039 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.415960 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:31.416210 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416162 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:31.416210 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416179 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\") pod \"c3194315-82b4-414a-9f65-73a1afa9efb6\" (UID: \"c3194315-82b4-414a-9f65-73a1afa9efb6\") " Oct 30 23:04:31.416328 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416277 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:04:31.416607 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416581 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.416607 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416608 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3194315-82b4-414a-9f65-73a1afa9efb6-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.416847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416623 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.416847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416639 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.416847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.416653 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.418154 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.418111 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:04:31.418259 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.418179 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:04:31.418259 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.418199 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:31.418353 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.418329 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg" pod "c3194315-82b4-414a-9f65-73a1afa9efb6" (UID: "c3194315-82b4-414a-9f65-73a1afa9efb6"). InnerVolumeSpecName "kube-api-access-bn9mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:04:31.517464 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.517433 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.517464 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.517459 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.517464 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.517469 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9mg\" (UniqueName: \"kubernetes.io/projected/c3194315-82b4-414a-9f65-73a1afa9efb6-kube-api-access-bn9mg\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:31.517714 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:31.517479 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3194315-82b4-414a-9f65-73a1afa9efb6-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:32.234116 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:32.234084 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" Oct 30 23:04:32.234116 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:32.234098 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8" event={"ID":"c3194315-82b4-414a-9f65-73a1afa9efb6","Type":"ContainerDied","Data":"1bd30dc6125f7f81be458cbb04fb08879116007d76eb427dfee6f4b6205508d5"} Oct 30 23:04:32.234647 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:32.234156 4305 scope.go:122] "RemoveContainer" containerID="17012ecc3ded6083b78f3c37219abe119269f1677b5daddd5563f7c142bb524b" Oct 30 23:04:32.434570 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:32.434535 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:04:32.438602 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:32.438573 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2e092e-openshift-default-86469f4d57n95q8"] Oct 30 23:04:33.387041 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387006 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:04:33.387441 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387409 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:04:33.387441 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387420 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:04:33.387441 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387439 4305 cpu_manager.go:414] "containerMap: removing container" podUID="69443d43-fdca-4e17-980d-63e07633d75b" containerName="maas-api" Oct 30 23:04:33.387546 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387445 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69443d43-fdca-4e17-980d-63e07633d75b" containerName="maas-api" Oct 30 23:04:33.387546 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387510 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="69443d43-fdca-4e17-980d-63e07633d75b" containerName="maas-api" Oct 30 23:04:33.387546 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.387525 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:04:33.391893 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.391873 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.486566 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.486533 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4bcacaaa-kserve-self-signed-certs\"" Oct 30 23:04:33.488404 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.488383 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:04:33.517304 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.517278 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" path="/var/lib/kubelet/pods/c3194315-82b4-414a-9f65-73a1afa9efb6/volumes" Oct 30 23:04:33.537229 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wld\" (UniqueName: \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.537356 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537234 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.537356 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537340 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.537479 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537460 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.537534 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537491 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.537534 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.537511 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.638986 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.638903 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.638986 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.638989 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639217 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639009 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639217 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639025 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639217 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639076 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wld\" (UniqueName: \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639217 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639094 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639499 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639472 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639563 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639536 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.639603 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.639581 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.641333 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.641312 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.641592 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.641576 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.696974 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.696943 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wld\" (UniqueName: \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\") pod \"e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:33.719242 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:33.719208 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:34.122040 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.122005 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:04:34.128345 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:04:34.128316 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4f84a2_975c_4697_9f4c_d2966a2483de.slice/crio-34eadba85bf846c8193bed8b760270a318486bf56dc6f548b52e80bb234e4b88 WatchSource:0}: Error finding container 34eadba85bf846c8193bed8b760270a318486bf56dc6f548b52e80bb234e4b88: Status 404 returned error can't find the container with id 34eadba85bf846c8193bed8b760270a318486bf56dc6f548b52e80bb234e4b88 Oct 30 23:04:34.244501 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.244472 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" event={"ID":"4c4f84a2-975c-4697-9f4c-d2966a2483de","Type":"ContainerStarted","Data":"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1"} Oct 30 23:04:34.244663 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.244510 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" event={"ID":"4c4f84a2-975c-4697-9f4c-d2966a2483de","Type":"ContainerStarted","Data":"34eadba85bf846c8193bed8b760270a318486bf56dc6f548b52e80bb234e4b88"} Oct 30 23:04:34.354143 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.354091 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" podStartSLOduration=1.354076726 podStartE2EDuration="1.354076726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:04:34.353021338 +0000 UTC m=+3794.948109659" watchObservedRunningTime="2026-10-30 23:04:34.354076726 +0000 UTC m=+3794.949165032" Oct 30 23:04:34.719590 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.719552 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:34.823617 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.823545 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:04:34.952048 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.952015 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\") pod \"419b82e5-fc69-462f-b436-c004b984ac4f\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " Oct 30 23:04:34.952239 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.952111 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\" (UniqueName: \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\") pod \"419b82e5-fc69-462f-b436-c004b984ac4f\" (UID: \"419b82e5-fc69-462f-b436-c004b984ac4f\") " Oct 30 23:04:34.952497 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.952467 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume" pod "419b82e5-fc69-462f-b436-c004b984ac4f" (UID: "419b82e5-fc69-462f-b436-c004b984ac4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:04:34.954291 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:34.954263 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k" pod "419b82e5-fc69-462f-b436-c004b984ac4f" (UID: "419b82e5-fc69-462f-b436-c004b984ac4f"). InnerVolumeSpecName "kube-api-access-4jh8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:04:35.053995 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.053962 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/419b82e5-fc69-462f-b436-c004b984ac4f-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:35.053995 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.053989 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jh8k\" (UniqueName: \"kubernetes.io/projected/419b82e5-fc69-462f-b436-c004b984ac4f-kube-api-access-4jh8k\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:35.250604 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.250568 4305 generic.go:356] "Generic (PLEG): container finished" podID="419b82e5-fc69-462f-b436-c004b984ac4f" containerID="558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22" exitCode=0 Oct 30 23:04:35.250847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.250615 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" Oct 30 23:04:35.250847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.250652 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" event={"ID":"419b82e5-fc69-462f-b436-c004b984ac4f","Type":"ContainerDied","Data":"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22"} Oct 30 23:04:35.250847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.250730 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r" event={"ID":"419b82e5-fc69-462f-b436-c004b984ac4f","Type":"ContainerDied","Data":"c555f96000a3697a27b05350246e93afdd8fcfe135bfb9729874a33e4e330406"} Oct 30 23:04:35.250847 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.250761 4305 scope.go:122] "RemoveContainer" containerID="558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22" Oct 30 23:04:35.261151 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.261128 4305 scope.go:122] "RemoveContainer" containerID="558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22" Oct 30 23:04:35.261451 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:04:35.261431 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22\": container with ID starting with 558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22 not found: ID does not exist" containerID="558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22" Oct 30 23:04:35.261555 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.261462 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22"} err="failed to get container status \"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22\": rpc error: code = NotFound desc = could not find container \"558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22\": container with ID starting with 558afdcf5935c40498dc15f5c22b757f40fa7d49978bd0e9c2bf090ac085ec22 not found: ID does not exist" Oct 30 23:04:35.403751 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.403714 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:04:35.412207 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.412179 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2e092e-57799b8f4rl29r"] Oct 30 23:04:35.517100 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:35.517029 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" path="/var/lib/kubelet/pods/419b82e5-fc69-462f-b436-c004b984ac4f/volumes" Oct 30 23:04:43.719703 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:43.719670 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:43.732209 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:43.732185 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:44.319071 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:04:44.319042 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:04:46.109186 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:04:46.109144 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:05:16.109167 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:05:16.109127 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:05:46.109008 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:05:46.108968 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:05:52.389128 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:05:52.389090 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:06:07.409041 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.408983 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" podUID="4c4f84a2-975c-4697-9f4c-d2966a2483de" containerName="main" containerID="cri-o://6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1" gracePeriod=45 Oct 30 23:06:07.668776 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.668711 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:06:07.693296 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693265 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\" (UniqueName: \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693460 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693310 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693460 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693334 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693460 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693398 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693627 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693520 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693627 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693564 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\") pod \"4c4f84a2-975c-4697-9f4c-d2966a2483de\" (UID: \"4c4f84a2-975c-4697-9f4c-d2966a2483de\") " Oct 30 23:06:07.693627 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693599 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:07.693786 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693639 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:07.693786 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693728 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:07.693967 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693945 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-model-cache\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.694024 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693974 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-tmp-dir\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.694024 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.693989 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-home\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.695753 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.695723 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:06:07.695907 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.695885 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:07.695971 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.695947 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld" pod "4c4f84a2-975c-4697-9f4c-d2966a2483de" (UID: "4c4f84a2-975c-4697-9f4c-d2966a2483de"). InnerVolumeSpecName "kube-api-access-b6wld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:06:07.726404 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.726380 4305 generic.go:356] "Generic (PLEG): container finished" podID="4c4f84a2-975c-4697-9f4c-d2966a2483de" containerID="6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1" exitCode=0 Oct 30 23:06:07.726528 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.726429 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" Oct 30 23:06:07.726528 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.726457 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" event={"ID":"4c4f84a2-975c-4697-9f4c-d2966a2483de","Type":"ContainerDied","Data":"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1"} Oct 30 23:06:07.726528 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.726498 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m" event={"ID":"4c4f84a2-975c-4697-9f4c-d2966a2483de","Type":"ContainerDied","Data":"34eadba85bf846c8193bed8b760270a318486bf56dc6f548b52e80bb234e4b88"} Oct 30 23:06:07.726528 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.726515 4305 scope.go:122] "RemoveContainer" containerID="6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1" Oct 30 23:06:07.735286 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.735268 4305 scope.go:122] "RemoveContainer" containerID="6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1" Oct 30 23:06:07.735564 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:06:07.735543 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1\": container with ID starting with 6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1 not found: ID does not exist" containerID="6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1" Oct 30 23:06:07.735641 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.735576 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1"} err="failed to get container status \"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1\": rpc error: code = NotFound desc = could not find container \"6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1\": container with ID starting with 6ee293a0055ec0e07330ce33355e00039c474d8b2deb15a1cb163ea0f064b0c1 not found: ID does not exist" Oct 30 23:06:07.795168 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.795145 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c4f84a2-975c-4697-9f4c-d2966a2483de-tls-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.795168 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.795166 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c4f84a2-975c-4697-9f4c-d2966a2483de-dshm\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.795310 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.795177 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wld\" (UniqueName: \"kubernetes.io/projected/4c4f84a2-975c-4697-9f4c-d2966a2483de-kube-api-access-b6wld\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:07.884415 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.884332 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:06:07.890984 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:07.890949 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4bcacaaa-kserve-cc7c5bff7-tzs2m"] Oct 30 23:06:09.525026 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:09.524734 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4f84a2-975c-4697-9f4c-d2966a2483de" path="/var/lib/kubelet/pods/4c4f84a2-975c-4697-9f4c-d2966a2483de/volumes" Oct 30 23:06:16.109166 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:06:16.109130 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:06:20.003380 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:20.003344 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:06:20.011578 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:20.011558 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:06:46.108549 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:06:46.108463 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:06:53.201412 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201369 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-68g4t"] Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201758 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4c4f84a2-975c-4697-9f4c-d2966a2483de" containerName="main" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201768 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4f84a2-975c-4697-9f4c-d2966a2483de" containerName="main" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201830 4305 cpu_manager.go:414] "containerMap: removing container" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201837 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201846 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201852 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201913 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3194315-82b4-414a-9f65-73a1afa9efb6" containerName="istio-proxy" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201922 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="419b82e5-fc69-462f-b436-c004b984ac4f" containerName="payload-processing" Oct 30 23:06:53.201998 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.201933 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4f84a2-975c-4697-9f4c-d2966a2483de" containerName="main" Oct 30 23:06:53.204888 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.204870 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.301812 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.301757 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-68g4t"] Oct 30 23:06:53.334718 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.334680 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e9a47-893e-4a9b-86b9-8308b7f4c701-openshift-service-ca\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.334900 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.334729 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-maas-api-tls\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.334900 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.334847 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-metrics-certs\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.334900 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.334874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/360e9a47-893e-4a9b-86b9-8308b7f4c701-kube-api-access-dkz8m\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.435578 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.435541 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-metrics-certs\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.435578 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.435578 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/360e9a47-893e-4a9b-86b9-8308b7f4c701-kube-api-access-dkz8m\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.435779 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.435659 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e9a47-893e-4a9b-86b9-8308b7f4c701-openshift-service-ca\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.435779 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.435695 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-maas-api-tls\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.436335 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.436303 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e9a47-893e-4a9b-86b9-8308b7f4c701-openshift-service-ca\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.438074 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.438041 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-metrics-certs\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.438220 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.438193 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e9a47-893e-4a9b-86b9-8308b7f4c701-maas-api-tls\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.493391 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.493305 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz8m\" (UniqueName: \"kubernetes.io/projected/360e9a47-893e-4a9b-86b9-8308b7f4c701-kube-api-access-dkz8m\") pod \"maas-api-f7b57794c-68g4t\" (UID: \"360e9a47-893e-4a9b-86b9-8308b7f4c701\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.520918 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.520885 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:53.891105 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.891009 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-68g4t"] Oct 30 23:06:53.896905 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:06:53.896874 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360e9a47_893e_4a9b_86b9_8308b7f4c701.slice/crio-0ce30676160bf8174ae112b16bf135c2ca75d3ed69e9d61afd3da1f74713eb06 WatchSource:0}: Error finding container 0ce30676160bf8174ae112b16bf135c2ca75d3ed69e9d61afd3da1f74713eb06: Status 404 returned error can't find the container with id 0ce30676160bf8174ae112b16bf135c2ca75d3ed69e9d61afd3da1f74713eb06 Oct 30 23:06:53.898262 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:53.898241 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:06:54.021264 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:54.021229 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" event={"ID":"360e9a47-893e-4a9b-86b9-8308b7f4c701","Type":"ContainerStarted","Data":"0ce30676160bf8174ae112b16bf135c2ca75d3ed69e9d61afd3da1f74713eb06"} Oct 30 23:06:55.027400 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:55.027357 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" event={"ID":"360e9a47-893e-4a9b-86b9-8308b7f4c701","Type":"ContainerStarted","Data":"a563990be2b2863ebe2612b8aebb1b458ddbcc0a101fee364be5d2b99cd81e6f"} Oct 30 23:06:55.027857 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:55.027505 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:06:55.138065 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:06:55.137958 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" podStartSLOduration=1.202301515 podStartE2EDuration="2.137937988s" totalImagesPullingTime="935.636473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:55.137899051 +0000 UTC m=+3935.732987357" watchObservedRunningTime="2026-10-30 23:06:55.137937988 +0000 UTC m=+3935.733026296" Oct 30 23:07:01.038086 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:01.038060 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f7b57794c-68g4t" Oct 30 23:07:01.196970 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:01.196938 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 23:07:01.197147 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:01.197123 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" podUID="6a52cc8e-17e1-4725-b642-ccae28dd7693" containerName="maas-api" containerID="cri-o://8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6" gracePeriod=30 Oct 30 23:07:02.446529 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.446500 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 23:07:02.628766 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.628697 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\" (UniqueName: \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\") pod \"6a52cc8e-17e1-4725-b642-ccae28dd7693\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " Oct 30 23:07:02.628926 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.628792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\") pod \"6a52cc8e-17e1-4725-b642-ccae28dd7693\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " Oct 30 23:07:02.628926 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.628837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\") pod \"6a52cc8e-17e1-4725-b642-ccae28dd7693\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " Oct 30 23:07:02.628926 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.628867 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\") pod \"6a52cc8e-17e1-4725-b642-ccae28dd7693\" (UID: \"6a52cc8e-17e1-4725-b642-ccae28dd7693\") " Oct 30 23:07:02.629218 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.629186 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca" pod "6a52cc8e-17e1-4725-b642-ccae28dd7693" (UID: "6a52cc8e-17e1-4725-b642-ccae28dd7693"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:02.631031 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.631002 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs" pod "6a52cc8e-17e1-4725-b642-ccae28dd7693" (UID: "6a52cc8e-17e1-4725-b642-ccae28dd7693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:02.631125 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.631060 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr" pod "6a52cc8e-17e1-4725-b642-ccae28dd7693" (UID: "6a52cc8e-17e1-4725-b642-ccae28dd7693"). InnerVolumeSpecName "kube-api-access-mntlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:02.631125 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.631059 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls" pod "6a52cc8e-17e1-4725-b642-ccae28dd7693" (UID: "6a52cc8e-17e1-4725-b642-ccae28dd7693"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:02.730430 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.730397 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:02.730430 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.730424 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a52cc8e-17e1-4725-b642-ccae28dd7693-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:02.730430 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.730433 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntlr\" (UniqueName: \"kubernetes.io/projected/6a52cc8e-17e1-4725-b642-ccae28dd7693-kube-api-access-mntlr\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:02.730642 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:02.730444 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a52cc8e-17e1-4725-b642-ccae28dd7693-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:03.064765 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.064730 4305 generic.go:356] "Generic (PLEG): container finished" podID="6a52cc8e-17e1-4725-b642-ccae28dd7693" containerID="8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6" exitCode=0 Oct 30 23:07:03.064957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.064776 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" event={"ID":"6a52cc8e-17e1-4725-b642-ccae28dd7693","Type":"ContainerDied","Data":"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6"} Oct 30 23:07:03.064957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.064780 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" Oct 30 23:07:03.064957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.064830 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72" event={"ID":"6a52cc8e-17e1-4725-b642-ccae28dd7693","Type":"ContainerDied","Data":"ae62c965dab0664dc68f1941ee835e8f83bc005dd46117c49da74c9ab5815c88"} Oct 30 23:07:03.064957 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.064849 4305 scope.go:122] "RemoveContainer" containerID="8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6" Oct 30 23:07:03.074572 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.074549 4305 scope.go:122] "RemoveContainer" containerID="8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6" Oct 30 23:07:03.074861 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:03.074841 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6\": container with ID starting with 8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6 not found: ID does not exist" containerID="8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6" Oct 30 23:07:03.074951 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.074875 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6"} err="failed to get container status \"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6\": rpc error: code = NotFound desc = could not find container \"8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6\": container with ID starting with 8031fd9e8936de56acb024aed7d9bb4c76ee5d57c3026b76454e48f0b82f28f6 not found: ID does not exist" Oct 30 23:07:03.219081 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.219047 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 23:07:03.265304 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.265276 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-cww72"] Oct 30 23:07:03.516863 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:03.516823 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a52cc8e-17e1-4725-b642-ccae28dd7693" path="/var/lib/kubelet/pods/6a52cc8e-17e1-4725-b642-ccae28dd7693/volumes" Oct 30 23:07:07.568867 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.568831 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:07.569338 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.569319 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6a52cc8e-17e1-4725-b642-ccae28dd7693" containerName="maas-api" Oct 30 23:07:07.569448 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.569341 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a52cc8e-17e1-4725-b642-ccae28dd7693" containerName="maas-api" Oct 30 23:07:07.569448 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.569423 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a52cc8e-17e1-4725-b642-ccae28dd7693" containerName="maas-api" Oct 30 23:07:07.573836 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.573817 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.623597 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.623565 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e41e55b-serving-cert\"" Oct 30 23:07:07.671669 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.671632 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:07.673206 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.673180 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.673269 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.673216 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.673608 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.673562 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.673762 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.673625 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxqx\" (UniqueName: \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.775021 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.774983 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.775231 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.775027 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxqx\" (UniqueName: \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.775231 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.775068 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.775231 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.775092 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.775231 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:07.775186 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-serving-cert: secret "maas-api-e2e-ait-6e41e55b-serving-cert" not found Oct 30 23:07:07.775476 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:07.775279 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls podName:f45c4d57-c9fe-4632-847d-57c8f7918b8d nodeName:}" failed. No retries permitted until 2026-10-30 23:07:08.275255518 +0000 UTC m=+3948.870343805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls") pod "maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" (UID: "f45c4d57-c9fe-4632-847d-57c8f7918b8d") : secret "maas-api-e2e-ait-6e41e55b-serving-cert" not found Oct 30 23:07:07.775863 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.775841 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.777498 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.777478 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:07.836653 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:07.836584 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxqx\" (UniqueName: \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:08.192567 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.192535 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:08.196341 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.196318 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.246792 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.246752 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e41e55b\"" Oct 30 23:07:08.246986 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.246834 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e41e55b-dockercfg-mqv6d\"" Oct 30 23:07:08.259448 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.259424 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:08.279835 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.279782 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mdz\" (UniqueName: \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.280019 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.279849 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:08.280084 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.280054 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.282775 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.282746 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") pod \"maas-api-e2e-ait-6e41e55b-666886d894-w8kj4\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:08.381689 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.381649 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.381892 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.381717 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mdz\" (UniqueName: \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.382400 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.382377 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.440667 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.440628 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mdz\" (UniqueName: \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\") pod \"payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.486835 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.486723 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:08.510688 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.510657 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:08.895292 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.895203 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:08.903432 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:07:08.903405 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45c4d57_c9fe_4632_847d_57c8f7918b8d.slice/crio-ed4a8850b1cdc65c0fa3dff8b2129130e5c436f905ab39651f9bb99a54d2f06f WatchSource:0}: Error finding container ed4a8850b1cdc65c0fa3dff8b2129130e5c436f905ab39651f9bb99a54d2f06f: Status 404 returned error can't find the container with id ed4a8850b1cdc65c0fa3dff8b2129130e5c436f905ab39651f9bb99a54d2f06f Oct 30 23:07:08.909777 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:08.909753 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:08.939280 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:07:08.939252 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23ac98e_95ca_43c7_95be_7f7d89c1c124.slice/crio-489449d9946e2b4e6a8482e795fac511c52041cd37924dfa104b43886bcc0ad7 WatchSource:0}: Error finding container 489449d9946e2b4e6a8482e795fac511c52041cd37924dfa104b43886bcc0ad7: Status 404 returned error can't find the container with id 489449d9946e2b4e6a8482e795fac511c52041cd37924dfa104b43886bcc0ad7 Oct 30 23:07:09.104164 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:09.104125 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" event={"ID":"f45c4d57-c9fe-4632-847d-57c8f7918b8d","Type":"ContainerStarted","Data":"ed4a8850b1cdc65c0fa3dff8b2129130e5c436f905ab39651f9bb99a54d2f06f"} Oct 30 23:07:09.105903 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:09.105864 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" event={"ID":"c23ac98e-95ca-43c7-95be-7f7d89c1c124","Type":"ContainerStarted","Data":"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016"} Oct 30 23:07:09.106059 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:09.105917 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" event={"ID":"c23ac98e-95ca-43c7-95be-7f7d89c1c124","Type":"ContainerStarted","Data":"489449d9946e2b4e6a8482e795fac511c52041cd37924dfa104b43886bcc0ad7"} Oct 30 23:07:09.106059 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:09.106004 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:09.216196 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:09.216134 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" podStartSLOduration=1.216116831 podStartE2EDuration="1.216116831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:09.214514775 +0000 UTC m=+3949.809603082" watchObservedRunningTime="2026-10-30 23:07:09.216116831 +0000 UTC m=+3949.811205141" Oct 30 23:07:10.112379 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:10.112294 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" event={"ID":"f45c4d57-c9fe-4632-847d-57c8f7918b8d","Type":"ContainerStarted","Data":"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a"} Oct 30 23:07:10.224333 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:10.224278 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" podStartSLOduration=2.266862628 podStartE2EDuration="3.224265066s" totalImagesPullingTime="957.402438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:10.222895014 +0000 UTC m=+3950.817983324" watchObservedRunningTime="2026-10-30 23:07:10.224265066 +0000 UTC m=+3950.819353372" Oct 30 23:07:11.116515 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:11.116477 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:16.108702 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:16.108664 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:07:17.126774 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:17.126745 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:20.115475 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:20.115443 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:46.109419 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:46.109374 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:07:47.241045 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:47.241009 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:52.241995 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.241949 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" podUID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" containerName="payload-pre-processing" containerID="cri-o://d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016" gracePeriod=25 Oct 30 23:07:52.484959 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.484931 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:52.519166 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.519089 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\") pod \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " Oct 30 23:07:52.519166 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.519128 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\" (UniqueName: \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\") pod \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\" (UID: \"c23ac98e-95ca-43c7-95be-7f7d89c1c124\") " Oct 30 23:07:52.519450 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.519421 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume" pod "c23ac98e-95ca-43c7-95be-7f7d89c1c124" (UID: "c23ac98e-95ca-43c7-95be-7f7d89c1c124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:52.521408 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.521372 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz" pod "c23ac98e-95ca-43c7-95be-7f7d89c1c124" (UID: "c23ac98e-95ca-43c7-95be-7f7d89c1c124"). InnerVolumeSpecName "kube-api-access-54mdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:52.620696 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.620655 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23ac98e-95ca-43c7-95be-7f7d89c1c124-plugins-config-volume\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:52.620696 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:52.620694 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mdz\" (UniqueName: \"kubernetes.io/projected/c23ac98e-95ca-43c7-95be-7f7d89c1c124-kube-api-access-54mdz\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:53.312128 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.312094 4305 generic.go:356] "Generic (PLEG): container finished" podID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" containerID="d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016" exitCode=0 Oct 30 23:07:53.312609 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.312143 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" Oct 30 23:07:53.312609 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.312167 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" event={"ID":"c23ac98e-95ca-43c7-95be-7f7d89c1c124","Type":"ContainerDied","Data":"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016"} Oct 30 23:07:53.312609 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.312207 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj" event={"ID":"c23ac98e-95ca-43c7-95be-7f7d89c1c124","Type":"ContainerDied","Data":"489449d9946e2b4e6a8482e795fac511c52041cd37924dfa104b43886bcc0ad7"} Oct 30 23:07:53.312609 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.312229 4305 scope.go:122] "RemoveContainer" containerID="d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016" Oct 30 23:07:53.320993 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.320976 4305 scope.go:122] "RemoveContainer" containerID="d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016" Oct 30 23:07:53.321285 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:53.321266 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016\": container with ID starting with d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016 not found: ID does not exist" containerID="d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016" Oct 30 23:07:53.321329 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.321292 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016"} err="failed to get container status \"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016\": rpc error: code = NotFound desc = could not find container \"d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016\": container with ID starting with d0d96615346e82858dbb3c9f6d46b27de20fb3809a8810a0f2e7214fc5895016 not found: ID does not exist" Oct 30 23:07:53.466380 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.466347 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:53.516761 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.516729 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e41e55b-7fc48d875f-nhrtj"] Oct 30 23:07:53.928041 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.927997 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:53.928331 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:53.928306 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" podUID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" containerName="maas-api" containerID="cri-o://54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a" gracePeriod=30 Oct 30 23:07:54.194388 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.194361 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:54.240580 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.240549 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\" (UniqueName: \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\") pod \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " Oct 30 23:07:54.240717 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.240588 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\") pod \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " Oct 30 23:07:54.240717 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.240699 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") pod \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " Oct 30 23:07:54.240791 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.240728 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\") pod \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\" (UID: \"f45c4d57-c9fe-4632-847d-57c8f7918b8d\") " Oct 30 23:07:54.241078 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.241050 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca" pod "f45c4d57-c9fe-4632-847d-57c8f7918b8d" (UID: "f45c4d57-c9fe-4632-847d-57c8f7918b8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:54.242872 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.242848 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx" pod "f45c4d57-c9fe-4632-847d-57c8f7918b8d" (UID: "f45c4d57-c9fe-4632-847d-57c8f7918b8d"). InnerVolumeSpecName "kube-api-access-pvxqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:54.242959 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.242884 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls" pod "f45c4d57-c9fe-4632-847d-57c8f7918b8d" (UID: "f45c4d57-c9fe-4632-847d-57c8f7918b8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:54.242959 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.242943 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs" pod "f45c4d57-c9fe-4632-847d-57c8f7918b8d" (UID: "f45c4d57-c9fe-4632-847d-57c8f7918b8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:54.320261 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.320209 4305 generic.go:356] "Generic (PLEG): container finished" podID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" containerID="54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a" exitCode=0 Oct 30 23:07:54.320261 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.320235 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" Oct 30 23:07:54.320866 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.320341 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" event={"ID":"f45c4d57-c9fe-4632-847d-57c8f7918b8d","Type":"ContainerDied","Data":"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a"} Oct 30 23:07:54.320866 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.320394 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4" event={"ID":"f45c4d57-c9fe-4632-847d-57c8f7918b8d","Type":"ContainerDied","Data":"ed4a8850b1cdc65c0fa3dff8b2129130e5c436f905ab39651f9bb99a54d2f06f"} Oct 30 23:07:54.320866 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.320429 4305 scope.go:122] "RemoveContainer" containerID="54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a" Oct 30 23:07:54.331836 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.331790 4305 scope.go:122] "RemoveContainer" containerID="54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a" Oct 30 23:07:54.332093 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:07:54.332074 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a\": container with ID starting with 54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a not found: ID does not exist" containerID="54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a" Oct 30 23:07:54.332150 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.332103 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a"} err="failed to get container status \"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a\": rpc error: code = NotFound desc = could not find container \"54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a\": container with ID starting with 54e7bc7e1189282534fa9be99d85d12f21282351952d9827e90951f9b847310a not found: ID does not exist" Oct 30 23:07:54.341670 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.341642 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxqx\" (UniqueName: \"kubernetes.io/projected/f45c4d57-c9fe-4632-847d-57c8f7918b8d-kube-api-access-pvxqx\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:54.341670 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.341668 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f45c4d57-c9fe-4632-847d-57c8f7918b8d-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:54.341870 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.341677 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:54.341870 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.341686 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f45c4d57-c9fe-4632-847d-57c8f7918b8d-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:54.474168 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.474137 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:54.477302 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:54.477278 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e41e55b-666886d894-w8kj4"] Oct 30 23:07:55.516896 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:55.516860 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" path="/var/lib/kubelet/pods/c23ac98e-95ca-43c7-95be-7f7d89c1c124/volumes" Oct 30 23:07:55.517293 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:55.517271 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" path="/var/lib/kubelet/pods/f45c4d57-c9fe-4632-847d-57c8f7918b8d/volumes" Oct 30 23:07:58.618999 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.618967 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:07:58.619388 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619348 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" containerName="payload-pre-processing" Oct 30 23:07:58.619388 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619358 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" containerName="payload-pre-processing" Oct 30 23:07:58.619388 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619373 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" containerName="maas-api" Oct 30 23:07:58.619388 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619378 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" containerName="maas-api" Oct 30 23:07:58.619521 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619445 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23ac98e-95ca-43c7-95be-7f7d89c1c124" containerName="payload-pre-processing" Oct 30 23:07:58.619521 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.619455 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45c4d57-c9fe-4632-847d-57c8f7918b8d" containerName="maas-api" Oct 30 23:07:58.622440 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.622413 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.671503 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.671470 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-152666-openshift-default-dockercfg-gp8s6\"" Oct 30 23:07:58.683108 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683062 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683270 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683162 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683315 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683282 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683358 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683326 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmhg\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683403 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683380 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683454 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683419 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683550 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683518 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683715 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683678 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.683871 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.683730 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.730598 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.730567 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:07:58.785121 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785314 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785135 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785314 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785178 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785314 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785210 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785314 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785298 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785560 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785332 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmhg\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785560 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785360 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785560 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785407 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.785560 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785450 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.786481 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.785929 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.786481 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.786065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.786481 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.786436 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.786733 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.786541 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.787185 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.787158 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.788613 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.788588 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.788870 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.788843 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.846765 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.846733 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmhg\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.847034 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.847013 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\") pod \"e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:58.939727 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:58.938865 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:07:59.371669 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:59.371591 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:07:59.377770 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:07:59.377745 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa569266_23fa_4d67_a8a6_8492fe1478b1.slice/crio-2867d9f0adc554ec8fa4924d96e6a10a5b68de0f0e0da5ce2bed521b7910b24f WatchSource:0}: Error finding container 2867d9f0adc554ec8fa4924d96e6a10a5b68de0f0e0da5ce2bed521b7910b24f: Status 404 returned error can't find the container with id 2867d9f0adc554ec8fa4924d96e6a10a5b68de0f0e0da5ce2bed521b7910b24f Oct 30 23:07:59.379953 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:59.379924 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:07:59.380050 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:59.379985 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:07:59.380050 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:07:59.380012 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:08:00.351368 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:00.351330 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerStarted","Data":"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b"} Oct 30 23:08:00.351754 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:00.351384 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerStarted","Data":"2867d9f0adc554ec8fa4924d96e6a10a5b68de0f0e0da5ce2bed521b7910b24f"} Oct 30 23:08:00.512264 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:00.512216 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podStartSLOduration=2.512201265 podStartE2EDuration="2.512201265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:00.51015896 +0000 UTC m=+4001.105247265" watchObservedRunningTime="2026-10-30 23:08:00.512201265 +0000 UTC m=+4001.107289571" Oct 30 23:08:00.939789 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:00.939755 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:00.944938 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:00.944914 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:01.356507 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:01.356430 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:01.357321 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:01.357300 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:09.311275 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.311182 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:08:09.315162 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.315138 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.364177 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.364137 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-152666-serving-cert\"" Oct 30 23:08:09.392479 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.392448 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w24v\" (UniqueName: \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.392646 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.392485 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.392646 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.392605 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.392770 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.392740 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.412844 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.412819 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:08:09.493838 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.493806 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w24v\" (UniqueName: \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.494017 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.493844 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.494017 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.493898 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.494017 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.493946 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.494017 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:08:09.494001 4305 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-serving-cert: secret "maas-api-e2e-authcfg-http-152666-serving-cert" not found Oct 30 23:08:09.494234 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:08:09.494075 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls podName:4c133b57-39da-4879-b1d6-e808d85fd2b3 nodeName:}" failed. No retries permitted until 2026-10-30 23:08:09.994054879 +0000 UTC m=+4010.589143177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls") pod "maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" (UID: "4c133b57-39da-4879-b1d6-e808d85fd2b3") : secret "maas-api-e2e-authcfg-http-152666-serving-cert" not found Oct 30 23:08:09.494665 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.494638 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.496285 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.496265 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.572620 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.572090 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w24v\" (UniqueName: \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:09.999599 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:09.999564 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:10.001968 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:10.001947 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:10.228514 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:10.228480 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:10.599552 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:10.599470 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:08:10.605867 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:08:10.605833 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c133b57_39da_4879_b1d6_e808d85fd2b3.slice/crio-9441ee04cc62708f5b8f9eacd53f6b7e493d8d1e8a8cbfec65ffb16a5fe851dc WatchSource:0}: Error finding container 9441ee04cc62708f5b8f9eacd53f6b7e493d8d1e8a8cbfec65ffb16a5fe851dc: Status 404 returned error can't find the container with id 9441ee04cc62708f5b8f9eacd53f6b7e493d8d1e8a8cbfec65ffb16a5fe851dc Oct 30 23:08:11.406666 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:11.406626 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" event={"ID":"4c133b57-39da-4879-b1d6-e808d85fd2b3","Type":"ContainerStarted","Data":"9441ee04cc62708f5b8f9eacd53f6b7e493d8d1e8a8cbfec65ffb16a5fe851dc"} Oct 30 23:08:12.413243 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:12.413203 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" event={"ID":"4c133b57-39da-4879-b1d6-e808d85fd2b3","Type":"ContainerStarted","Data":"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978"} Oct 30 23:08:12.413638 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:12.413359 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:12.526761 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:12.526708 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" podStartSLOduration=2.482490626 podStartE2EDuration="3.526693469s" totalImagesPullingTime="1.044202843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:12.525328119 +0000 UTC m=+4013.120416425" watchObservedRunningTime="2026-10-30 23:08:12.526693469 +0000 UTC m=+4013.121781775" Oct 30 23:08:16.108818 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:08:16.108763 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:08:18.423582 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:18.423547 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:08:26.514093 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.514054 4305 generic.go:356] "Generic (PLEG): container finished" podID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerID="287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b" exitCode=0 Oct 30 23:08:26.514528 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.514131 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerDied","Data":"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b"} Oct 30 23:08:26.514639 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.514613 4305 scope.go:122] "RemoveContainer" containerID="287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b" Oct 30 23:08:26.515891 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.515864 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:08:26.515973 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.515928 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:08:26.515973 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:26.515955 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 23:08:27.519913 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:27.519876 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerStarted","Data":"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1"} Oct 30 23:08:27.940030 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:27.939997 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:27.941270 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:27.941245 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 23:08:27.941350 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:27.941329 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 23:08:28.939697 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:28.939657 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 23:08:28.940162 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:28.939729 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 23:08:29.939554 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:29.939518 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 23:08:29.939720 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:29.939573 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 23:08:30.940349 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:30.940314 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 23:08:30.940750 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:30.940372 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 23:08:31.360878 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:31.359908 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:31.939981 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:31.939945 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 23:08:31.940173 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:31.940004 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 23:08:32.944518 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:32.944491 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:32.945393 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:08:32.945367 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:08:46.109358 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:08:46.109319 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:09:16.108974 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:09:16.108936 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:09:46.109080 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:09:46.109002 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:10:08.152891 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.152852 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:10:08.153430 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.153064 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerName="maas-api" containerID="cri-o://9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978" gracePeriod=30 Oct 30 23:10:08.424327 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.424305 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:10:08.537476 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.537444 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\") pod \"4c133b57-39da-4879-b1d6-e808d85fd2b3\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " Oct 30 23:10:08.537679 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.537518 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") pod \"4c133b57-39da-4879-b1d6-e808d85fd2b3\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " Oct 30 23:10:08.537679 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.537545 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\" (UniqueName: \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\") pod \"4c133b57-39da-4879-b1d6-e808d85fd2b3\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " Oct 30 23:10:08.537679 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.537594 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\") pod \"4c133b57-39da-4879-b1d6-e808d85fd2b3\" (UID: \"4c133b57-39da-4879-b1d6-e808d85fd2b3\") " Oct 30 23:10:08.538175 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.538154 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca" pod "4c133b57-39da-4879-b1d6-e808d85fd2b3" (UID: "4c133b57-39da-4879-b1d6-e808d85fd2b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:10:08.541202 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.541172 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs" pod "4c133b57-39da-4879-b1d6-e808d85fd2b3" (UID: "4c133b57-39da-4879-b1d6-e808d85fd2b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:10:08.541492 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.541461 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls" pod "4c133b57-39da-4879-b1d6-e808d85fd2b3" (UID: "4c133b57-39da-4879-b1d6-e808d85fd2b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:10:08.541848 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.541822 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v" pod "4c133b57-39da-4879-b1d6-e808d85fd2b3" (UID: "4c133b57-39da-4879-b1d6-e808d85fd2b3"). InnerVolumeSpecName "kube-api-access-5w24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:10:08.638691 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.638657 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-metrics-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:08.638691 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.638685 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c133b57-39da-4879-b1d6-e808d85fd2b3-maas-api-tls\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:08.638691 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.638695 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w24v\" (UniqueName: \"kubernetes.io/projected/4c133b57-39da-4879-b1d6-e808d85fd2b3-kube-api-access-5w24v\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:08.638944 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.638705 4305 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c133b57-39da-4879-b1d6-e808d85fd2b3-openshift-service-ca\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:08.984081 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.984051 4305 generic.go:356] "Generic (PLEG): container finished" podID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerID="9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978" exitCode=0 Oct 30 23:10:08.984263 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.984097 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" Oct 30 23:10:08.984263 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.984142 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" event={"ID":"4c133b57-39da-4879-b1d6-e808d85fd2b3","Type":"ContainerDied","Data":"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978"} Oct 30 23:10:08.984263 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.984200 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" event={"ID":"4c133b57-39da-4879-b1d6-e808d85fd2b3","Type":"ContainerDied","Data":"9441ee04cc62708f5b8f9eacd53f6b7e493d8d1e8a8cbfec65ffb16a5fe851dc"} Oct 30 23:10:08.984263 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.984228 4305 scope.go:122] "RemoveContainer" containerID="9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978" Oct 30 23:10:08.993463 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.993443 4305 scope.go:122] "RemoveContainer" containerID="9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978" Oct 30 23:10:08.993772 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:10:08.993736 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978\": container with ID starting with 9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978 not found: ID does not exist" containerID="9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978" Oct 30 23:10:08.993839 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:08.993781 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978"} err="failed to get container status \"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978\": rpc error: code = NotFound desc = could not find container \"9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978\": container with ID starting with 9887f37ac89aaae0cc15b79bdbc900d2c4db4c81b83427d54087891f40fde978 not found: ID does not exist" Oct 30 23:10:09.139649 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:09.139619 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:10:09.185270 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:09.185233 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn"] Oct 30 23:10:09.517082 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:09.517051 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" path="/var/lib/kubelet/pods/4c133b57-39da-4879-b1d6-e808d85fd2b3/volumes" Oct 30 23:10:11.419234 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:11.419186 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-152666-5f696bcc8b-jksrn" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.140:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 23:10:16.108282 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:10:16.108245 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:10:17.336587 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:17.336550 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:10:17.337052 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:17.336818 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" containerID="cri-o://7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1" gracePeriod=30 Oct 30 23:10:17.947902 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:17.947868 4305 patch_prober.go:29] interesting pod/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:10:17.948073 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:17.947919 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:10:22.578790 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.578767 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:10:22.694611 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694581 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.694756 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694620 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.694756 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694664 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.694756 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694696 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.694756 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694716 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.694756 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.694743 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.695088 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695064 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:10:22.695088 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695077 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.695220 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695088 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:10:22.695220 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695136 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.695220 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695153 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:10:22.695220 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695190 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\") pod \"fa569266-23fa-4d67-a8a6-8492fe1478b1\" (UID: \"fa569266-23fa-4d67-a8a6-8492fe1478b1\") " Oct 30 23:10:22.695427 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695408 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:10:22.695509 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695486 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:10:22.695736 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695716 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-certs\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.695849 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695740 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-data\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.695849 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695754 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa569266-23fa-4d67-a8a6-8492fe1478b1-istiod-ca-cert\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.695849 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695769 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-credential-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.695849 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.695784 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-workload-socket\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.697155 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.697126 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:10:22.697301 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.697260 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:10:22.697600 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.697567 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 23:10:22.697698 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.697576 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg" pod "fa569266-23fa-4d67-a8a6-8492fe1478b1" (UID: "fa569266-23fa-4d67-a8a6-8492fe1478b1"). InnerVolumeSpecName "kube-api-access-gzmhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:10:22.796842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.796789 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-token\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.796842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.796832 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-podinfo\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.796842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.796841 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa569266-23fa-4d67-a8a6-8492fe1478b1-istio-envoy\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:22.796842 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:22.796851 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmhg\" (UniqueName: \"kubernetes.io/projected/fa569266-23fa-4d67-a8a6-8492fe1478b1-kube-api-access-gzmhg\") on node \"ip-10-0-134-125.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:10:23.051476 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.051389 4305 generic.go:356] "Generic (PLEG): container finished" podID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerID="7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1" exitCode=0 Oct 30 23:10:23.051476 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.051449 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" Oct 30 23:10:23.051665 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.051471 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerDied","Data":"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1"} Oct 30 23:10:23.051665 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.051551 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8" event={"ID":"fa569266-23fa-4d67-a8a6-8492fe1478b1","Type":"ContainerDied","Data":"2867d9f0adc554ec8fa4924d96e6a10a5b68de0f0e0da5ce2bed521b7910b24f"} Oct 30 23:10:23.051665 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.051573 4305 scope.go:122] "RemoveContainer" containerID="7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1" Oct 30 23:10:23.060596 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.060579 4305 scope.go:122] "RemoveContainer" containerID="287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b" Oct 30 23:10:23.068239 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.068222 4305 scope.go:122] "RemoveContainer" containerID="7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1" Oct 30 23:10:23.068500 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:10:23.068479 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1\": container with ID starting with 7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1 not found: ID does not exist" containerID="7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1" Oct 30 23:10:23.068568 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.068513 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1"} err="failed to get container status \"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1\": rpc error: code = NotFound desc = could not find container \"7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1\": container with ID starting with 7b21f19c5fb42347adfd1490fa26d28a909e518129635356c8909a36c6f65ea1 not found: ID does not exist" Oct 30 23:10:23.068568 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.068538 4305 scope.go:122] "RemoveContainer" containerID="287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b" Oct 30 23:10:23.068830 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:10:23.068809 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b\": container with ID starting with 287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b not found: ID does not exist" containerID="287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b" Oct 30 23:10:23.068879 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.068834 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b"} err="failed to get container status \"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b\": rpc error: code = NotFound desc = could not find container \"287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b\": container with ID starting with 287c0f4e3e3bec8fccc33577aa8e88e28312471ed159f21bf860026145cf830b not found: ID does not exist" Oct 30 23:10:23.209547 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.209511 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:10:23.255492 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.255450 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-152666-openshift-default-7bc87b8cbf-rdvn8"] Oct 30 23:10:23.517652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:10:23.517617 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" path="/var/lib/kubelet/pods/fa569266-23fa-4d67-a8a6-8492fe1478b1/volumes" Oct 30 23:10:46.110079 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:10:46.110043 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:11:16.108740 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:11:16.108697 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:11:20.033566 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:20.033534 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:11:20.040101 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:20.040082 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-857zd_0d296dcb-2c18-45b4-9ae4-97fc339d9095/ovn-acl-logging/0.log" Oct 30 23:11:46.096300 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:46.096261 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zrwhd_dea49084-0340-4d8d-86b0-fc05b559a2f7/manager/0.log" Oct 30 23:11:46.111181 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:11:46.111146 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:11:46.891135 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:46.891103 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:11:47.884688 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:47.884659 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4pmf5_16f613be-f4da-466b-9e94-f068104933fe/proxy/0.log" Oct 30 23:11:48.682315 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:48.682285 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f7b57794c-68g4t_360e9a47-893e-4a9b-86b9-8308b7f4c701/maas-api/0.log" Oct 30 23:11:49.916065 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:49.916039 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fjg54_40c9a0b6-ea50-4947-bde5-cab611ff1779/postgres/0.log" Oct 30 23:11:54.396647 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:54.396613 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qsnv7_02a00e8d-f14d-4827-8a45-865982c1d36a/discovery/0.log" Oct 30 23:11:54.639812 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:54.639780 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784dfbd64d-7pfnm_5cda4a18-49ba-4716-b38e-7aa00c726785/kube-auth-proxy/0.log" Oct 30 23:11:55.121513 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:55.121486 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-4kprw_0f68ce5c-3d8d-418a-8ecf-563c339c1e0c/istio-proxy/0.log" Oct 30 23:11:55.607583 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:55.607555 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-cg4z5_a1ffecbd-ee8d-4e9f-a7bf-f4348f280830/istio-proxy/0.log" Oct 30 23:11:56.612411 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:56.612379 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-n92qw_982a22ec-5768-4a3c-843d-42d811121213/istio-proxy/0.log" Oct 30 23:11:59.311425 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:59.311396 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr_46b614aa-46bd-4c6e-8894-8758705961d8/main/0.log" Oct 30 23:11:59.605114 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:59.605042 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vgnc2_bed155d2-07c5-43c7-a6a9-0bc34a01e4ef/main/0.log" Oct 30 23:11:59.841604 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:11:59.841569 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-5blq9_7f8b3232-9131-40d2-81ba-b2c3f7ec6562/main/0.log" Oct 30 23:12:00.075907 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:00.075875 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rqm2c_06f21021-46c6-4539-b4e0-5ce82fb489b0/main/0.log" Oct 30 23:12:00.307719 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:00.307693 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv_1968f99b-e5ac-4169-861b-8a7d8c0787a8/main/0.log" Oct 30 23:12:00.545065 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:00.545039 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-v5lxk_cd65b990-64a6-433a-870e-f3e60fb09f37/main/0.log" Oct 30 23:12:12.235336 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:12.235278 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zrwhd_dea49084-0340-4d8d-86b0-fc05b559a2f7/manager/0.log" Oct 30 23:12:12.958200 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:12.958168 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:12:13.834531 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:13.834501 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4pmf5_16f613be-f4da-466b-9e94-f068104933fe/proxy/0.log" Oct 30 23:12:14.456192 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:14.456159 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f7b57794c-68g4t_360e9a47-893e-4a9b-86b9-8308b7f4c701/maas-api/0.log" Oct 30 23:12:15.732323 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:15.732292 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fjg54_40c9a0b6-ea50-4947-bde5-cab611ff1779/postgres/0.log" Oct 30 23:12:16.108187 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:12:16.108079 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:12:19.564566 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:19.564533 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qsnv7_02a00e8d-f14d-4827-8a45-865982c1d36a/discovery/0.log" Oct 30 23:12:19.834461 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:19.834378 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784dfbd64d-7pfnm_5cda4a18-49ba-4716-b38e-7aa00c726785/kube-auth-proxy/0.log" Oct 30 23:12:20.293775 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:20.293724 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-4kprw_0f68ce5c-3d8d-418a-8ecf-563c339c1e0c/istio-proxy/0.log" Oct 30 23:12:20.766042 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:20.766014 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-cg4z5_a1ffecbd-ee8d-4e9f-a7bf-f4348f280830/istio-proxy/0.log" Oct 30 23:12:21.762919 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:21.762886 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-n92qw_982a22ec-5768-4a3c-843d-42d811121213/istio-proxy/0.log" Oct 30 23:12:24.202501 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:24.202475 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ch5pr_46b614aa-46bd-4c6e-8894-8758705961d8/main/0.log" Oct 30 23:12:24.435634 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:24.435606 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vgnc2_bed155d2-07c5-43c7-a6a9-0bc34a01e4ef/main/0.log" Oct 30 23:12:24.667067 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:24.666986 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-5blq9_7f8b3232-9131-40d2-81ba-b2c3f7ec6562/main/0.log" Oct 30 23:12:24.899416 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:24.899390 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rqm2c_06f21021-46c6-4539-b4e0-5ce82fb489b0/main/0.log" Oct 30 23:12:25.135523 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:25.135494 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8mg7hv_1968f99b-e5ac-4169-861b-8a7d8c0787a8/main/0.log" Oct 30 23:12:25.371990 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:25.371957 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-v5lxk_cd65b990-64a6-433a-870e-f3e60fb09f37/main/0.log" Oct 30 23:12:46.108733 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:12:46.108647 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:12:55.439280 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:55.439248 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:12:56.026293 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:12:56.026261 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:13:16.108938 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:13:16.108899 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:13:35.084909 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:35.084873 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:13:35.668751 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:35.668717 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8cf787766-76dxg_15e59070-1878-47e8-9e79-8528184385f9/manager/0.log" Oct 30 23:13:46.108784 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:13:46.108750 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor" Oct 30 23:13:46.434809 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:46.434731 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qhvzk_e4971693-7a0b-46a6-80bc-ad5329f213a7/global-pull-secret-syncer/0.log" Oct 30 23:13:49.901497 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:49.901469 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5r47b_b1f1f8ea-fb7b-4bfb-be7b-cfe42a2ea4f3/konnectivity-agent/0.log" Oct 30 23:13:50.708717 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:50.708680 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-125.us-west-2.compute.internal_52151281fe9c0d516ce7c27b4aad2f39/haproxy/0.log" Oct 30 23:13:56.524858 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.524820 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g"] Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525187 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525197 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525208 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerName="maas-api" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525213 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerName="maas-api" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525224 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525229 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525297 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525304 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c133b57-39da-4879-b1d6-e808d85fd2b3" containerName="maas-api" Oct 30 23:13:56.525389 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.525313 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa569266-23fa-4d67-a8a6-8492fe1478b1" containerName="istio-proxy" Oct 30 23:13:56.528477 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.528458 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.577502 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.577474 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"openshift-service-ca.crt\"" Oct 30 23:13:56.577649 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.577499 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"kube-root-ca.crt\"" Oct 30 23:13:56.623759 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.623724 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2xpx\"/\"default-dockercfg-xvgbt\"" Oct 30 23:13:56.625668 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.625645 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g"] Oct 30 23:13:56.648448 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.648424 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-proc\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.648553 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.648454 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bwb\" (UniqueName: \"kubernetes.io/projected/d2c937d8-f0a2-4231-aff9-3a392367ee24-kube-api-access-l8bwb\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.648553 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.648478 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-sys\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.648644 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.648584 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-podres\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.648644 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.648633 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-lib-modules\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749538 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-proc\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749544 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bwb\" (UniqueName: \"kubernetes.io/projected/d2c937d8-f0a2-4231-aff9-3a392367ee24-kube-api-access-l8bwb\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749567 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-sys\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749621 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-podres\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749652 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749628 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-proc\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749853 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-lib-modules\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749853 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749694 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-podres\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749853 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749712 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-sys\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.749853 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.749762 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c937d8-f0a2-4231-aff9-3a392367ee24-lib-modules\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.808404 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.808351 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bwb\" (UniqueName: \"kubernetes.io/projected/d2c937d8-f0a2-4231-aff9-3a392367ee24-kube-api-access-l8bwb\") pod \"perf-node-gather-daemonset-g268g\" (UID: \"d2c937d8-f0a2-4231-aff9-3a392367ee24\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:56.840646 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:56.840615 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:57.208479 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:57.208446 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g"] Oct 30 23:13:57.215051 ip-10-0-134-125 kubenswrapper[4305]: W1030 23:13:57.215021 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2c937d8_f0a2_4231_aff9_3a392367ee24.slice/crio-c203a8f28a1a3e25b442e170df52eae02517175db4622dc0176ec662ada6a8aa WatchSource:0}: Error finding container c203a8f28a1a3e25b442e170df52eae02517175db4622dc0176ec662ada6a8aa: Status 404 returned error can't find the container with id c203a8f28a1a3e25b442e170df52eae02517175db4622dc0176ec662ada6a8aa Oct 30 23:13:58.078593 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:58.078556 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" event={"ID":"d2c937d8-f0a2-4231-aff9-3a392367ee24","Type":"ContainerStarted","Data":"2733c76daa07e93b7246d0fd67fb477bc9300f5f819929356528e0c43da4ccba"} Oct 30 23:13:58.079111 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:58.078612 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" event={"ID":"d2c937d8-f0a2-4231-aff9-3a392367ee24","Type":"ContainerStarted","Data":"c203a8f28a1a3e25b442e170df52eae02517175db4622dc0176ec662ada6a8aa"} Oct 30 23:13:58.079111 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:58.078679 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:13:58.189082 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:13:58.189027 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" podStartSLOduration=2.189009145 podStartE2EDuration="2.189009145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:58.188522734 +0000 UTC m=+4358.783611044" watchObservedRunningTime="2026-10-30 23:13:58.189009145 +0000 UTC m=+4358.784097454" Oct 30 23:14:04.093477 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:04.093400 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-g268g" Oct 30 23:14:04.835352 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:04.835321 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7l86_29ad770b-01c4-4963-a910-1a5010e4e79c/kube-state-metrics/0.log" Oct 30 23:14:05.253223 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:05.253196 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7l86_29ad770b-01c4-4963-a910-1a5010e4e79c/kube-rbac-proxy-main/0.log" Oct 30 23:14:05.677095 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:05.677027 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w7l86_29ad770b-01c4-4963-a910-1a5010e4e79c/kube-rbac-proxy-self/0.log" Oct 30 23:14:06.514771 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:06.514740 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-l7s7p_1ca3d1a3-da6b-4d82-9f8f-eb5da422a26d/monitoring-plugin/0.log" Oct 30 23:14:06.943410 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:06.943384 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cg66d_6d2c6300-9238-44b0-910b-ce103e181d4a/node-exporter/0.log" Oct 30 23:14:07.371994 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:07.371928 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cg66d_6d2c6300-9238-44b0-910b-ce103e181d4a/kube-rbac-proxy/0.log" Oct 30 23:14:07.796339 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:07.796315 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cg66d_6d2c6300-9238-44b0-910b-ce103e181d4a/init-textfile/0.log" Oct 30 23:14:10.803213 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:10.803187 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xnpcn_ebf74d8f-5bae-4c22-851c-564d30ef8088/kube-rbac-proxy-main/0.log" Oct 30 23:14:11.226038 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:11.226005 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xnpcn_ebf74d8f-5bae-4c22-851c-564d30ef8088/kube-rbac-proxy-self/0.log" Oct 30 23:14:11.651463 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:11.651388 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xnpcn_ebf74d8f-5bae-4c22-851c-564d30ef8088/openshift-state-metrics/0.log" Oct 30 23:14:14.985283 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:14.985255 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r2jvg_a54829c0-8734-4647-9589-b9046fbe56a5/prometheus-operator/0.log" Oct 30 23:14:15.403975 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:15.403901 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r2jvg_a54829c0-8734-4647-9589-b9046fbe56a5/kube-rbac-proxy/0.log" Oct 30 23:14:15.828324 ip-10-0-134-125 kubenswrapper[4305]: I1030 23:14:15.828293 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8cz8j_f55dcc39-82dd-4403-bee1-38b004c540d7/prometheus-operator-admission-webhook/0.log" Oct 30 23:14:16.108762 ip-10-0-134-125 kubenswrapper[4305]: E1030 23:14:16.108677 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hsrj5" containerName="extractor"